Skip to content

Hypothesis Test Power Calibration

Design a hypothesis test around the effect that would actually matter, then tune sample size, noise control, allocation, and error rates so the test has adequate power to detect it.

The Diagnostic Story

Symptom: A study is planned around the budget and timeline rather than around the effect size that would actually matter. Null results are treated as reassurance even though the design was never able to detect a meaningful difference. The analysis specifies how unlikely a false positive must be, but no one ever asked how likely the test is to miss a real effect. Resources are spent confirming nothing when the test was never sensitive enough to say anything useful.

Pivot: Before treating a hypothesis test as decision-ready, specify the minimum effect size that would have practical significance, choose acceptable false-positive and false-negative risks, model the relevant noise and design assumptions, and adjust sample size, allocation, measurement precision, or variance controls until the test reaches its target operating characteristics.

Resolution: Studies avoid being launched when they cannot answer the question. Null results carry explicit sensitivity bounds so decision-makers know what size of effect was ruled out, not merely undetected. Resources concentrate on the design choices that most improve inferential value.

Reach for this when you hear…

[clinical trials] “We need 80% power to detect a 10-point symptom reduction — if we can only enroll 40 patients we shouldn't start the trial.”

[product A/B testing] “We called it no effect after two weeks, but our minimum detectable effect was 15% and we were looking for a 3% lift.”

[policy evaluation] “The program evaluation came back null, but given their sample size they couldn't have detected the effect even if it was twice what we hoped for.”

Mechanisms / Implementations

  • Closed-Form Power Calculation
  • Minimum Detectable Effect Table
  • Operating Characteristic Curve
  • Pilot Variance Estimation
  • Power Sensitivity Grid
  • Pre-Analysis Power Statement
  • Simulation-Based Power Analysis

Abstractions this archetype builds on — directly (a source ingredient) or as a related pattern. Links follow the typed catalog namespace.

Built directly on (2)

  • Statistical Inference: Reasoning from a finite, noisy sample back to the underlying population or process while explicitly quantifying the uncertainty that sampling introduces.
  • Statistical Power: Probability of detecting effect.

Also references 12 related abstractions

Variants

Narrower or domain-specific specializations that share this archetype's core structure. Recognized variants are established; candidate variants are provisional.

Analytical Power Calculation Variant · mechanism family variant · recognized

Use closed-form statistical formulas to choose sample size or estimate power for a standard hypothesis test.

Simulation-Based Power Calibration Variant · mechanism family variant · recognized

Estimate power by repeatedly simulating data under plausible true effects and applying the planned analysis procedure.

Minimum Detectable Effect Planning Variant · communication variant · recognized

Present the effect size that the proposed design can detect, then decide whether that sensitivity is substantively adequate.

Attrition-Adjusted Power Calibration Variant · risk or failure variant · recognized

Calibrate design sensitivity after expected dropout, nonresponse, missing observations, or unusable measurements are accounted for.