Multiple Testing Discipline

Control false discoveries when many comparisons, claims, or tests are being tried.

The Diagnostic Story

Symptom: A result looks significant, or a pattern looks real, but the search that found it involved dozens of comparisons, subgroups, metrics, or time windows that nobody is counting. The one finding that got presented was selected because it was the most interesting — not because it was pre-specified. Failed comparisons disappear from the record, so the claimed discovery looks far more credible than the actual search process warrants. When someone tries to reproduce it, the pattern is gone.

Pivot: Define the family of related tests or claims being made, track the full search process rather than just the selected result, and choose a decision rule that accounts for how many chances existed for an accidental pattern. Separate exploratory evidence — leads worth investigating — from confirmatory evidence that is ready to justify action.

Resolution: The rate of false discoveries is bounded because the opportunity set that produced any selected finding remains visible and is factored into interpretation. High-stakes action requires confirmation stronger than the initial search result, so the distinction between a lead and a finding is preserved rather than collapsed. The discovery record becomes more credible because null results and failed comparisons are retained alongside the claimed success.

Reach for this when you hear…

[clinical research] “We ran thirty-two subgroup analyses and found one that was significant — that's not a finding, that's what chance looks like when you look often enough.”

[A/B testing] “The dashboard shows a green metric and everyone wants to ship, but we've been peeking at this experiment every day for two weeks and we never preregistered a stopping rule.”

[financial audit] “If we flag every anomaly that crosses a threshold after slicing by region, quarter, and account type, we'll bury the real findings in false positives — we need a family-level rule before we start slicing.”

Mechanisms / Implementations

• Bonferroni-Like Correction: A strict threshold-adjustment mechanism for contexts where even one false positive across the family is costly.
• False Discovery Rate Control: A screening mechanism that permits many discoveries while controlling the expected share that are false.
• Preregistration: A procedural commitment mechanism that records planned claims and analyses before results are observed.
• Holdout Validation: An evidence-partitioning mechanism that reserves fresh data or cases for confirmation after discovery.
• Claim Registry: A governance mechanism that tracks attempted claims, statuses, owners, and follow-up requirements.
• Replication Study: An independent-confirmation mechanism that tests whether a selected finding persists in new evidence.
• Confirmatory Follow-Up: A staged-validation mechanism that turns a promising lead into a targeted confirmatory test.
• Alpha-Spending Plan: A sequential-testing mechanism for repeated interim looks.
• Metric Hierarchy: A priority mechanism that prevents teams from promoting a secondary metric after the primary result disappoints.
• Multiverse Analysis Report: A transparency mechanism that shows whether a result depends on one favorable analytic path.

Related Abstractions

Abstractions this archetype builds on — directly (a source ingredient) or as a related pattern. Links follow the typed catalog namespace.

Built directly on (3)

• Multiple Comparisons Correction: Adjust the thresholds or p-values of a defined family of simultaneous tests so a chosen family-level error criterion remains bounded despite multiplicity.
• Reproducibility & Replicability: Repeatable results.
• Type I & Type II Errors: False positive/negative.

Also references 6 related abstractions

• Confirmation Bias: Favor confirming evidence.
• Hypothesis Testing (Null vs. Alternative): Null vs alternative evaluation.
• Probability: Quantifies uncertainty and likelihoods.
• Statistical Significance (p-Value): Likelihood results are random.
• Threshold: Safe vs harmful levels.
• Uncertainty: Incomplete knowledge.

Variants

Narrower or domain-specific specializations that share this archetype's core structure. Recognized variants are established; candidate variants are provisional.

Familywise Error Control Variant · risk or failure variant · recognized

A stricter variant that tries to avoid even one false positive across a defined family of claims.

False Discovery Rate Variant · risk or failure variant · recognized

A screening-oriented variant that permits many discoveries while controlling the expected share that are false.

Exploratory–Confirmatory Partition Variant · temporal variant · promote to full archetype candidate

A staged variant that explicitly labels idea-generation evidence separately from confirmation evidence.

Claim Registry Variant · governance variant · recognized

A governance variant that controls multiplicity by making attempted claims, status, and follow-up obligations visible across a portfolio.