Skip to content

Hypothesis Testing Frame

Frame a claim against a default alternative so evidence can change belief or action under explicit error risks.

The Diagnostic Story

Symptom: Decisions about whether evidence is convincing happen after the result is known, and the threshold keeps shifting to match whoever is arguing. A null finding becomes proof of absence, while a significant finding becomes proof of importance and causality without checking whether the design could actually support those claims. Metrics get swapped mid-analysis for ones that gave a better-looking number. The process produces a verdict but not a trustworthy one.

Pivot: Fix the claim under test, the default position, the alternative, the evidence threshold, and the false-positive and false-negative costs before observing results. Define the decision rule so interpretation cannot move after the outcome is visible.

Resolution: Evidence is evaluated against a stable standard rather than a retrospective one. Post-hoc threshold shopping and burden-shifting become visible violations rather than normal practice. A result's meaning is bounded — inconclusive evidence stays distinguishable from proof of no effect, and a significant result stays distinguishable from proof of causality or practical importance.

Reach for this when you hear…

[drug approval] “You don't get to decide your primary endpoint after unblinding — that ship sailed when you filed the protocol.”

[fraud investigation] “The threshold for what counts as suspicious keeps moving based on who the suspect is, and that means the standard itself is the problem.”

[engineering reliability] “We need to agree before the test run whether a 5% failure rate is a pass or a fail — I'm not deciding it after I see the number.”

Mechanisms / Implementations

  • Null Hypothesis Significance Test: A formal statistical mechanism for evaluating a default/null claim against an alternative under assumptions and thresholds.
  • Decision Threshold Rule: A procedural mechanism that turns evidence into a status change.
  • Quality Acceptance Test: An operational mechanism for accepting, rejecting, or reworking a batch, product, service, or process.
  • A/B Test Interpretation Protocol: A product or service experimentation mechanism that interprets measured differences against planned metrics and launch criteria.
  • Legal Burden-of-Proof Analog: A governance mechanism that assigns a default presumption and evidence burden before changing a status or imposing a consequence.
  • Inspection Pass/Fail Test: A classification mechanism for checking whether an item meets criteria.
  • Falsification Protocol: A mechanism for stating what would count against a favored claim before evidence is reviewed.
  • Equivalence or Noninferiority Test: A mechanism for showing that a difference is small enough or that a new option is not unacceptably worse.
  • Sequential Review Gate: A workflow mechanism for interpreting evidence at predefined milestones.
  • Scientific Claim Evaluation Template: A template mechanism that prompts users to document claim, default, alternative, evidence, threshold, assumptions, error costs, and limits.

Abstractions this archetype builds on — directly (a source ingredient) or as a related pattern. Links follow the typed catalog namespace.

Built directly on (3)

Also references 8 related abstractions

Variants

Narrower or domain-specific specializations that share this archetype's core structure. Recognized variants are established; candidate variants are provisional.

Null–Alternative Statistical Test Variant · mechanism family variant · recognized

A formal statistical variant that evaluates a null/default hypothesis against an alternative using a specified test statistic and threshold.

Superiority Test Variant · subtype · recognized

A variant where the alternative claims that one option, process, treatment, or policy performs better than the default or comparator.

Equivalence / Noninferiority Variant · subtype · recognized

A variant where the aim is to show that a new option is close enough to, or not unacceptably worse than, an existing standard.

Burden-of-Proof Variant · governance variant · recognized

A governance-oriented variant that assigns which side must provide evidence and what standard must be met before a presumption changes.

Falsification Frame Variant · risk or failure variant · recognized

A variant that defines in advance what observation would count against a favored claim so evidence search does not become one-sided confirmation.