Bonferroni-Like Correction¶

Statistical threshold adjustment — instantiates Multiple-Testing Discipline

Stiffens each test's significance bar in proportion to how many tests share the family, so that clearing it stays hard even after many simultaneous attempts.

Version: v1 · 2026-08-24 · History
Mechanism #: 866
Type: Statistical Threshold Adjustment
Form family: Analysis, Modeling & Optimization
Solution family: Evidence, Inference & Validation
Problem family: Uncertainty, Evidence & Inference Failure
Problem subfamily: Experimental Comparison & Hypothesis-Test Design
Origin domain: Statistics & Experimental Design
Also from: Mathematics
Instantiates: Multiple-Testing Discipline

A Bonferroni-Like Correction is the simplest way to keep a whole family of comparisons honest: decide how many tests belong together, then make the bar for each individual test proportionally stricter so that the chance of any false alarm across the entire family stays under the target. If you run twenty independent comparisons each at the usual 5% bar, you should expect one to "hit" by chance alone; the correction fixes this by requiring each test to clear 0.05 divided by twenty (or an equivalent step-down variant) instead. The one idea that makes it this mechanism and not a sibling: it protects against even a single false positive anywhere in the family — the familywise error rate — and it does so by dividing one error budget across the breadth of many distinct, simultaneous tests, using nothing more than the family's headcount.

Example¶

A collaboration hunting for a new subatomic particle scans a collider's data for an unexpected bump in the mass spectrum. But they do not test one mass — they sweep hundreds of narrow mass windows across the searchable range, and some window will show an upward fluctuation purely by chance. A bump that looks like a rare 1-in-3-million fluctuation (roughly the "5-sigma" convention physicists demand) is far less impressive once you remember it is the most extreme of, say, five hundred windows examined. This is the look-elsewhere effect.^[1] The team applies a trials-factor correction: they multiply the local significance by the number of independent windows searched, converting the eye-catching local p-value into a "global" one that accounts for the whole scanned family. A bump that was locally 5-sigma might survive as global 4.2-sigma — still interesting, worth more data — or it might collapse to noise. The correction is what keeps the collaboration from announcing a discovery that is really just the luckiest of many looks.

How it works¶

Define the family. Decide which comparisons must be protected together — every window, metric, or contrast that had a chance to produce the headline.
Count the attempts. Tally the number of tests in that family; this count is the entire input to the correction.
Divide the budget. Set each test's threshold to the target familywise rate divided by the count (Bonferroni), or apply a step-down ordering (Holm) that is uniformly less conservative while preserving the same familywise guarantee.
Judge against the stiffened bar. A result counts only if it clears its corrected threshold; borderline findings that would have passed the naïve bar are held back. The guarantee rests on Boole's inequality — the probability of any event in a set is at most the sum of their individual probabilities — which is why the method needs no assumptions about how the tests relate.

Tuning parameters¶

Family boundary — what counts as "the family." Drawing it wide (every window ever examined) is strict; drawing it narrow (only pre-specified windows) is more powerful but easier to game.
Target familywise rate — the total tolerable probability of any false positive; lower is safer but costs sensitivity across the board.
Step-down vs single-step — a Holm-style ordering recovers power over plain division at no cost to the guarantee, at the price of a slightly less transparent rule.
Correlation handling — plain Bonferroni assumes the worst and is over-conservative when tests are correlated; a permutation-based effective count can loosen it when many windows overlap.

When it helps, and when it misleads¶

Its strength is bluntness: it needs only the family size, makes no assumption about dependence among tests, and gives a guarantee that is easy to state and to audit — exactly the properties you want when even one false claim is dangerous or embarrassing. That makes it the natural choice for confirmatory settings with a modest, well-defined family.

Its failure mode is that the same worst-case blindness that makes it robust also makes it savagely conservative when tests are many or correlated: divide 0.05 across ten thousand comparisons and almost no real effect can clear the bar, so genuine signals are suppressed — overcorrection paralysis. The classic misuse is the mirror image: shrinking the family after the fact to only the tests that survived, so the "correction" is applied to a family gerrymandered around the winner. The guarding discipline is to fix the family's boundary before looking and to reach for a discovery-rate procedure instead when the family is large and follow-up is cheap.

How it implements the components¶

multiplicity_adjustment_rule — the division of the error budget across the family is the adjustment rule in its most literal form.
claim_family — the correction cannot begin until the set of comparisons that must be protected together is named; defining that family is half of what this mechanism does.
multiplicity_inventory — the tally of how many tests are in the family is the exact quantity the threshold is divided by.

It does not relax the standard to permit many discoveries while capping their false share; setting that tolerance through an error_risk_profile and stamping each surviving hypothesis with a result_status_label is False Discovery Rate Control's job, not this strict familywise rule's.

Instantiates: Multiple-Testing Discipline — supplies the strict familywise-correction form of multiplicity control.
Sibling mechanisms: Alpha-Spending Plan · False Discovery Rate Control · Claim Registry · Confirmatory Follow-Up · Holdout Validation · Metric Hierarchy · Multiverse Analysis Report · Preregistration · Replication Study

Editorial Notes¶

Form Classification¶

Form family: Analysis, Modeling & Optimization

Rationale: Stiffens each test's significance bar in proportion to how many tests share the family, so that clearing it stays hard even after many simultaneous attempts, making its operative form a computation or analytic transformation that produces an inference, comparison, or optimized result.

Independent corroboration: The frozen evidence defines Bonferroni-Like Correction as 'Stiffens each test's significance bar in proportion to how many tests share the family, so that clearing it stays hard even after many simultaneous attempts', so its operative form is Analysis, Modeling & Optimization.

Review outcome: Independent reviewer agreement; high confidence.

Origin Attribution¶

Primary origin: Statistics & Experimental Design

Origin pattern: Single lineage

Present-day reach: Specialized

Rationale: Bonferroni and Holm procedures divide or step down a familywise error budget so the probability of any false positive across a declared test family stays bounded.

Related originating lineages:

Mathematics — Mathematics contributes the formal structure, proof, asymptotic, combinatorial, or numerical foundation used here.