Multiple-Testing Review¶
Statistical review — instantiates Pattern Detection with Validation
Audits how many patterns were searched before one looked meaningful, then raises the evidence bar to match the size of that search — while watching that the correction does not go so far it buries the real effects.
A Multiple-Testing Review asks a question no single result can answer about itself: how many patterns did you look at before this one lit up? When a search scans thousands of candidate associations, the most striking one will almost always clear a naïve significance bar by chance alone — significance was priced for one test, not ten thousand. This review reconstructs the true size of the search and re-prices the evidence threshold accordingly, so a "finding" has to be surprising relative to how hard it was looked for. Its defining move is treating the multiplicity of the search itself as the thing under suspicion. But it is not a one-way skeptic: over-correct and every real but modest effect is thrown out with the noise, so the review holds the false-positive tax and the false-negative cost in the same hand.
Example¶
A lab runs a genome-wide association study, testing roughly a million genetic variants for association with a trait. Twelve variants come back with p-values below the conventional 0.05, and the excited first draft names all twelve. The Multiple-Testing Review starts from the number that matters: a million independent tests at p < 0.05 would be expected to throw up tens of thousands of "hits" by chance, so twelve is not a triumph — it is fewer than noise predicts. The review applies the field's genome-wide significance threshold (roughly 5×10⁻⁸, a bar set precisely because a million tests were run), and eleven of the twelve evaporate. The one survivor cleared a bar sized to the search.
Then the review turns the other way. A variant that missed the corrected bar but sits in a gene already implicated by independent biology is not simply "negative" — the blanket correction may have cost a real signal. The review flags it for replication rather than burial. The output is not "one true hit and eleven fakes" but a re-priced landscape: what survives a search-sized bar, and what the correction may have wrongly silenced.
How it works¶
What distinguishes this review from an ordinary significance test is that it audits the process that produced the candidate, not just the candidate:
- Reconstruct the search space. Count every comparison actually made — including the ones that were run and quietly dropped — because the effective number of tests, not the number reported, sets the real false-positive rate.
- Re-price the threshold. Raise the bar to control the family-wise error rate (a Bonferroni-style correction) or the expected proportion of false discoveries among the hits (a false-discovery-rate control), matching the correction to the size of the search.
- Check the false-positive side. Ask whether the surviving pattern could still be a selection artifact — subgroups sliced until one was significant, outcomes swapped, the "garden of forking paths" of analyst choices.
- Check the false-negative side. Ask what the correction cost: which modest-but-real effects the search-sized bar just suppressed, and which deserve replication rather than dismissal.
Tuning parameters¶
- Error-rate target — control the family-wise error rate (strict, conservative) or the false-discovery rate (more powerful, tolerates a stated fraction of false hits); the choice sets how much real signal you trade for cleanliness.
- Effective test count — whether correlated tests are counted as one or many; treating dependent tests as independent over-corrects and needlessly buries findings.
- Pre-registration weight — how much a hypothesis specified before the search counts against one found during it; strong weighting sharply cuts false discovery but constrains exploration.
- Replication bar — whether survivors must reappear in an independent dataset before they are believed at all — the strongest correction of the lot.
- Salvage threshold — how much prior support a corrected-away candidate needs before it is flagged for replication instead of dropped, tuning the false-negative guard.
When it helps, and when it misleads¶
Its strength is that it neutralizes the single most reliable way to manufacture a false pattern — searching until something crosses a fixed line — and it does so with an explicit, defensible re-pricing rather than a hunch.[1] It is indispensable wherever discovery is wide and cheap: high-throughput biology, A/B-test dashboards with dozens of metrics, quantitative backtesting, sensor fleets.
Its failure modes come in two directions. Under-correction lets the largest of many random bumps parade as a discovery — the review's whole reason to exist. Over-correction is the opposite sin: a blanket Bonferroni across correlated tests can crush every real but subtle effect, converting a fishing expedition into a false all-clear. The classic misuse is applying the correction after peeking — choosing the threshold once the hits are known, or reporting the search as if only the surviving test had ever been run. The guarding discipline is to fix the correction and the search definition before the results are seen, count dependent tests honestly, and route corrected-away-but-plausible candidates to replication rather than to the bin.
How it implements the components¶
evidence_threshold— it recomputes the bar the pattern must clear, sizing it to the number of comparisons made rather than to a single test.false_positive_review— its central act is checking whether the apparent pattern is a product of search volume: a selection artifact surviving because so many candidates were tried.false_negative_review— it explicitly weighs the power cost of correction, flagging real-but-modest effects the search-sized bar may have wrongly suppressed.
It never names or specifies the pattern itself and never checks it against prevalence — writing the explicit candidate and its base_rate_context is Diagnostic Pattern Checklist — and it confirms nothing on a fresh sample: testing survivors on later independent data (validation_sample) is Trend Validation Review. This review interrogates the size and honesty of the search.
Related¶
- Instantiates: Pattern Detection with Validation — it supplies the search-multiplicity correction that keeps a wide hunt from certifying its luckiest bump.
- Sibling mechanisms: Diagnostic Pattern Checklist · Recurrence Tracking Dashboard · System Archetype Matching · Trend Validation Review · Held-Out Sample Test · Signal/Noise Review · Pattern Library · Base Rate Check
Editorial Notes¶
Form Classification¶
Form family: Assessment, Review & Assurance
Rationale: Multiple-Testing Review operates as a bounded evaluation of existing evidence or work that produces a finding or disposition because it audits how many patterns were searched before one looked meaningful, then raises the evidence bar to match the size of that search — while watching that the correction does not go so far it buries the real effects.
Independent corroboration: The frozen evidence defines Multiple-Testing Review as 'Audits how many patterns were searched before one looked meaningful, then raises the evidence bar to match the size of that search — while watching that the correction does not go so far it buries the real effects', so its operative form is Assessment, Review & Assurance.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Statistics & Experimental Design
Origin pattern: Single lineage
Present-day reach: Multi-domain
Rationale: Auditing the number of searched hypotheses and correcting the evidence threshold with FDR or related procedures is canonical statistical practice.
Related originating lineages:
- Data Science & Analytics — High-dimensional pattern search and researcher degrees of freedom make the review central to applied analytics.
Review resolution: Both independent reviews agree on primary origin statistics_experimental_design; reconciliation resolves secondary fields (alternate_origin_disagreement). Alternate origins retained (data_science) are the union of reviewer-supported formative lineages with explicit rationales, not a list of later application domains. Present-day breadth is represented separately as domain_reach=multi_domain; origin_mode=single_lineage records the historical relationship among lineages. Confidence is conservatively reconciled to high, and encyclopedia_synthesis=false preserves either reviewer's finding that the encyclopedia generalized the mechanism.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
Multiple-Testing Review and Held-Out Sample Test are the two classic overfitting guards, and they attack it from opposite ends: the held-out test asks whether a fitted pattern survives on data it never touched, while this review asks whether the pattern would even look real once you account for how many were tried. A candidate can pass one and fail the other — a hit that replicates out of sample may still have been cherry-picked from a huge search, and vice versa — so a rigorous pipeline applies both.
References¶
[1] The false discovery rate — the expected proportion of false positives among declared discoveries — and the Benjamini–Hochberg procedure that controls it (1995) are the standard tools for re-pricing significance under many comparisons; they trade a small, stated fraction of false hits for far more power than the stricter Bonferroni bound. withdrawn registry ↩