Hypothesis Testing Frame¶
Frame a claim against a default alternative so evidence can change belief or action under explicit error risks.
The Diagnostic Story¶
Symptom: Decisions about whether evidence is convincing happen after the result is known, and the threshold keeps shifting to match whoever is arguing. A null finding becomes proof of absence, while a significant finding becomes proof of importance and causality without checking whether the design could actually support those claims. Metrics get swapped mid-analysis for ones that gave a better-looking number. The process produces a verdict but not a trustworthy one.
Pivot: Fix the claim under test, the default position, the alternative, the evidence threshold, and the false-positive and false-negative costs before observing results. Define the decision rule so interpretation cannot move after the outcome is visible.
Resolution: Evidence is evaluated against a stable standard rather than a retrospective one. Post-hoc threshold shopping and burden-shifting become visible violations rather than normal practice. A result's meaning is bounded — inconclusive evidence stays distinguishable from proof of no effect, and a significant result stays distinguishable from proof of causality or practical importance.
Reach for this when you hear…¶
[drug approval] “You don't get to decide your primary endpoint after unblinding — that ship sailed when you filed the protocol.”
[fraud investigation] “The threshold for what counts as suspicious keeps moving based on who the suspect is, and that means the standard itself is the problem.”
[engineering reliability] “We need to agree before the test run whether a 5% failure rate is a pass or a fail — I'm not deciding it after I see the number.”
When This Archetype Applies¶
Partial catalog groundingSome structural conditions are represented by existing abstractions, but no sufficient condition set is fully represented.
Diagnostic problem
A claim is assessed informally or after-the-fact, allowing ambiguous evidence thresholds, moving standards, one-sided interpretation, and confusion between evidence, action, causality, and practical importance.
What this problem means
The structural problem is not merely lack of data. It is the absence of a stable interpretation frame. A team sees evidence, then decides what it was testing. A reviewer accepts a claim because the result looks impressive, without saying what default was displaced. A non-detection is treated as proof of no effect even though the evidence may have been too weak to detect anything. A threshold is chosen because it supports a preferred conclusion.
Without the frame, claim evaluation becomes vulnerable to ambiguity, confirmation bias, burden shifting, threshold shopping, and overclaiming. The same evidence can be used to support incompatible conclusions because the claim, default, threshold, and error costs were never fixed.
Show the applicability expression
Applicability expression4 distinct conditions
groundedpartly groundedopen
4 conditions, all required.
4Required in every casenumbered 1–4
These hold no matter which pattern applies.
Claim disposition decision · grounded
A decision depends on accepting, rejecting, escalating, or leaving a claim unresolved.
The source archetype describes the situation as follows: A decision depends on whether a claim should be accepted, rejected, escalated, or left unresolved. The normalized requirement above isolates the load-bearing portion used in this condition set.
Ambiguous probabilistic evidence · grounded
Evidence is noisy, partial, probabilistic, contested, or open to multiple interpretations.
The system needs to learn from evidence, but evidence can be interpreted opportunistically unless the claim, default, threshold, and error risks are fixed before interpretation. The narrower requirement in this condition set is: Evidence is noisy, partial, probabilistic, contested, or open to multiple interpretations.
Post-evidence threshold manipulation · 2 cases · 0 matched
Actors can move thresholds after seeing evidence or select the result supporting a preferred conclusion.
The source archetype describes the situation as follows: Actors can move thresholds after observing evidence or emphasize whichever result supports their preferred conclusion. The normalized requirement above isolates the load-bearing portion used in this condition set.
Protected default presumption · grounded
A default presumption should not change without a clear evidentiary burden.
The source archetype describes the situation as follows: A default presumption or incumbent state should not change without a clear evidentiary burden. The normalized requirement above isolates the load-bearing portion used in this condition set.
Other requirements and context (2)
Why these sit outside the expression
Supporting context — it may accompany or help interpret the situation, but it is not a load-bearing condition in a sufficient diagnostic set.
Deployment constraint — it constrains how the intervention must be deployed, not the situation that calls for it.
Supporting contextFalse positives and false negatives have materially different costs.
It is especially useful when a result could be interpreted opportunistically, when actors might move standards after seeing evidence, or when the two major mistakes—false acceptance and false rejection—carry different costs. In this archetype, the relevant contextual consideration is: False positives and false negatives have materially different costs. It helps interpret the situation or strengthens the practical case for examining the archetype.
Deployment constraintA result must be communicated with limits rather than converted into simplistic proof or disproof.
Coverage
3 of 4 conditions grounded · 1 open.
Mechanisms / Implementations¶
- Null Hypothesis Significance Test: Implements a default-versus-alternative comparison with a formal statistical threshold under specified assumptions.
- Decision Threshold Rule: Operationalizes the evidence threshold as a cut point, burden, gate, or standard that changes action status.
- Quality Acceptance Test: Uses predefined acceptance criteria to decide whether a product, batch, process, or deliverable meets a required standard.
- A/B Test Interpretation Protocol: Reads a live randomized experiment against a pre-declared primary metric and launch criteria, turning the measured difference into a ship, hold, or iterate decision.
- Legal Burden-of-Proof Analog: Uses a formal presumption and evidentiary burden to protect against costly false judgments.
- Inspection Pass/Fail Test: Applies predefined criteria to classify an item, process, or condition as acceptable or unacceptable.
- Falsification Protocol: Specifies what evidence would count against a favored claim before the evidence is sought.
- Equivalence or Noninferiority Test: Implements a variant where the goal is to show sufficiently small difference or no unacceptable loss rather than superiority.
- Sequential Review Gate: Re-evaluates evidence at predefined milestones while controlling how interim findings change action.
- Scientific Claim Evaluation Template: Prompts analysts to state claim, default, alternative, evidence, assumptions, thresholds, error costs, and interpretation limits.
Related Abstractions¶
Abstractions this archetype builds on — directly (a source ingredient) or as a related pattern. Links follow the typed catalog namespace.
Built directly on (3)
- Hypothesis Testing (Null vs. Alternative): Null vs alternative evaluation.
- Statistical Significance (p-Value): Likelihood results are random.
- Type I & Type II Errors: False positive/negative.
Also references 8 related abstractions
- Bayesian Updating: Update beliefs with evidence.
- Effect Size: Magnitude of effect.
- Multiple Comparisons Correction: Adjust the thresholds or p-values of a defined family of simultaneous tests so a chosen family-level error criterion remains bounded despite multiplicity.
- Probability: Quantifies uncertainty and likelihoods.
- Reproducibility & Replicability: Repeatable results.
- Statistical Power: Probability of detecting effect.
- Threshold: Safe vs harmful levels.
- Uncertainty: Incomplete knowledge.
Variants¶
Narrower or domain-specific specializations that share this archetype's core structure. Recognized variants are established; candidate variants are provisional.
Null–Alternative Statistical Test Variant · mechanism family variant · recognized
A formal statistical variant that evaluates a null/default hypothesis against an alternative using a specified test statistic and threshold.
Superiority Test Variant · subtype · recognized
A variant where the alternative claims that one option, process, treatment, or policy performs better than the default or comparator.
Equivalence / Noninferiority Variant · subtype · recognized
A variant where the aim is to show that a new option is close enough to, or not unacceptably worse than, an existing standard.
Burden-of-Proof Variant · governance variant · recognized
A governance-oriented variant that assigns which side must provide evidence and what standard must be met before a presumption changes.
Falsification Frame Variant · risk or failure variant · recognized
A variant that defines in advance what observation would count against a favored claim so evidence search does not become one-sided confirmation.
Editorial Notes¶
Problem Classification¶
Classification: Uncertainty, Evidence & Inference Failure → Experimental Comparison & Hypothesis-Test Design
Problem kernel: claim testing lacks precommitted hypotheses and error standards
Rationale: After-the-fact thresholds and moving interpretations blur evidence, action, causal import, and practical significance.
Independent corroboration: The earliest necessary condition in the frozen evidence is: A claim is assessed informally or after-the-fact, allowing ambiguous evidence thresholds, moving standards, one-sided interpretation, and confusion between evidence, action, causality, and practical importance. That is a experimental comparison and hypothesis test design problem because Treatment, control, assignment, blinding, power, and evidence thresholds are insufficiently designed to support the intended comparison.
Review outcome: Independent reviewer agreement; high confidence.