Statistical Power¶

Prime #: 437
Origin domain: Statistics & Experimental Design
Aliases: Power Analysis, Sensitivity Analysis Statistical, 1 Minus Beta, Detection Probability
Related primes: Hypothesis Testing (Null vs. Alternative), Type I & Type II Errors, Effect Size, Statistical Significance (p-Value), Confidence Intervals, Sampling (Representativeness), Reproducibility & Replicability, Randomization

Core Idea¶

Statistical power is the detection-probability-for-true-effects principle that quantifies the probability a hypothesis test will correctly reject a false null hypothesis. Formally, power = P(reject H₀ | H₁ is true), equivalently 1−β where β is the Type II error probability^[1]. Power is a function of four quantities bound by a rigid mathematical relationship: the effect size δ (the true magnitude of effect under H₁), the sample size n (or equivalently the number of experimental units and allocation to groups), the significance level α (the pre-specified Type I error rate), and the variability σ (the noise in outcome measurement)—any three of these plus power determine the fourth, and power analysis is the systematic computation that makes these relationships explicit for study planning and interpretation. The concept emerged from the Neyman-Pearson 1933 framework for hypothesis testing, which explicitly formalized power as the complement of Type II error and introduced the notion of the most powerful test at a given α. Jacob Cohen's 1962 JASP paper "The statistical power of abnormal-social psychological research" and his subsequent 1969/1988 Statistical Power Analysis for the Behavioral Sciences^[2] established power analysis as a systematic practice in the social sciences, quantified conventional effect sizes (small, medium, large), and provided tables and computational tools that made power analysis accessible to non-statistician researchers. The modern practice is supported by dedicated software (G*Power, PASS, R packages pwr and WebPower, SAS PROC POWER) and by routine power-analysis requirements in grant applications and study protocols. Power analysis is the pre-specified planning discipline that prevents the twin failures of underpowering (running studies that cannot detect meaningful effects, producing inconclusive results and wasted resources) and overpowering (running studies much larger than needed, detecting trivial effects as significant while consuming excessive resources)^[2]; underpowered studies are the more common and more damaging failure mode—they produce inflated effect-size estimates when they do reach significance (because only the higher-variance realizations cross the threshold, a phenomenon called the "winner's curse" or type M error; Gelman-Carlin 2014), contribute to replication failures (the original underpowered study's inflated effect cannot be reproduced by adequately-powered replications), and waste resources on null-producing studies.

How would you explain it like I'm…

Catching Real Effects

Imagine looking for a small bug in the grass. With a tiny magnifying glass and bad lighting, you'll probably miss it even if it's right there. With a big magnifying glass and a bright lamp, you'll find it. Statistical power is just how good your bug-finder is at spotting a bug that really is there.

Chance of catching a real effect

Suppose you think a new sports drink helps kids run faster. To check, you test some kids. If you only test 3 kids, even if the drink really works, the small group might not show the difference — you'd miss it. If you test 300 kids, you're much more likely to spot it. Statistical power is the chance that your study will actually catch a real effect when one exists. Bigger effects, bigger samples, and less random noise all give you more power.

Effect-detection probability

Statistical power is the probability that a study will correctly detect a true effect — that is, reject a null hypothesis when the null is actually false. Formally, power equals 1 minus the chance of a false negative. Four things determine it: the size of the real effect, the size of your sample, the significance threshold you set, and how noisy your measurements are. If your study has low power, you might run it and find 'no effect' even when one really exists — wasting time and money. Worse, the few low-power studies that do find an effect tend to overstate it, because only the lucky high-variance results crossed the bar. That's why scientists do 'power analysis' before running studies: to figure out how big a sample they need to give themselves a fair shot at detecting what they're hoping to find.

Statistical power is the probability that a hypothesis test correctly rejects a false null hypothesis: power = P(reject H0 | H1 true) = 1 - beta, where beta is the Type II error rate. Power is a function of four interlocked quantities — true effect size (delta), sample size (n), significance level (alpha), and measurement noise (sigma) — bound by a deterministic relationship: fixing any three pins down the fourth, and power analysis is the systematic computation that makes this explicit at the design stage. The framework was formalized by Neyman and Pearson (1933), but its widespread adoption in the social sciences traces to Jacob Cohen's 1962 audit of abnormal-social psychology and his 1969/1988 *Statistical Power Analysis for the Behavioral Sciences*, which introduced the conventional small/medium/large effect-size benchmarks and produced the tabulations that made power calculations practical. The pre-specified planning discipline guards against two failure modes. Underpowering — running studies too small to detect plausible effects — wastes resources, yields inconclusive nulls, and worse, when an underpowered study does cross the significance threshold the resulting estimate is systematically inflated (the winner's curse, or type-M error; Gelman and Carlin 2014) because only the high-variance realizations were able to clear the bar. Overpowering wastes resources detecting trivially small effects. Underpowering is the dominant and more damaging failure, and a major contributor to the replication crisis.

Structural Signature¶

A power analysis exhibits: (a) a pre-specified test — the hypothesis test whose power is to be characterized — with its test statistic, null distribution, and significance level α; (b) an alternative hypothesis expressed as an effect-size value δ under which power is to be computed; © a probability model specifying the sampling distribution of the test statistic under H₁ (typically a non-central version of the null distribution, parameterized by the non-centrality parameter which depends on δ, n, and σ); (d) specification of n, α, δ, and σ — the four quantities that jointly determine power, with three fixed and one solved for; (e) a computational method — closed-form formula (t-test, F-test, chi-squared), tabulated value (Cohen's handbook), or simulation-based (complex designs); (f) design-specific adjustments — design-effect for cluster designs, allocation-ratio for unequal-n, multiplicity-adjustment for multiple primary outcomes; (g) sensitivity characterization — how power varies with effect-size assumptions (because the "true" δ is unknown, power at several plausible δ values is informative)^[3]; (h) reporting that documents all assumptions so the analysis can be reproduced and its sensitivity understood. When these elements are properly implemented, the power analysis supports informed study design and interpretation; when they are missing or pro-forma (generic "n = 30 per group for 80% power to detect a medium effect" without justifying the medium-effect assumption), the power analysis bears little relation to actual study performance.

What It Is Not¶

Not a post-hoc diagnostic of observed results. "Observed power" or "post-hoc power" computed using the observed effect size is a mechanical function of the observed p-value (p close to 0.05 → observed power close to 50%; p << 0.05 → observed power high) and provides no independent information beyond the p-value. The American Statistical Association and methodological literature have repeatedly criticized observed-power reporting. Legitimate retrospective analyses use pre-specified or external effect-size estimates to characterize what the study could have detected.
Not solely about sample size. Power depends on effect size, variability, α, and sample size jointly. Increasing sample size is one lever; reducing measurement variability, choosing more efficient tests, and relaxing α are others. Design efficiency (e.g., paired versus independent designs, blocking, covariate adjustment) can substantially increase power without changing n^[4].
Not a substitute for effect-size consideration. Power analysis presupposes a specified effect size (typically the minimum clinically or practically meaningful effect). Choosing the effect size is itself a scientific judgment that power analysis cannot avoid; using Cohen's conventional "medium" effect without domain-specific justification is common but problematic.
Not the probability that a significant result is true. This is the positive predictive value of the test (Ioannidis 2005), which depends on power, α, and the prior probability of a real effect. Power alone does not provide this value.
Not meaningful outside the pre-specified testing framework. Post-hoc data exploration, multiple unadjusted comparisons, and analysis selection all erode the nominal α, and hence the nominal power, of any single "primary" test. Power calculations assume the analysis will proceed as pre-specified.
Not independent of the probability model. Nonparametric tests, tests under model violations, and complex designs produce power that differs from naive normal-theory formulas. Simulation-based power is often required for non-standard situations.
Not constant across effect-size values. The power function rises from α at δ = 0 to 1 at large δ. "80% power" is a point on this curve, not a property of the test in general. Reporting power at multiple plausible effect-size values is more informative than a single point^[3].
Not automatically 80%. The 80% convention reflects a particular judgment about acceptable Type II error rate (20%) given typical Type I error rate (5%), implying a Type I:Type II ratio of 4:1. This ratio may or may not fit the decision context; some contexts warrant 90% or even 95% power.
Not unrelated to publication bias and winner's curse. Underpowered studies that reach significance tend to overestimate effect sizes; this inflation feeds publication bias (selective publication of "positive" underpowered studies) and contributes to replication failures.
Not sufficient by itself for study quality. A study can be adequately powered but have other problems (selection bias, confounding, measurement error, analysis flexibility) that undermine inference. Power is necessary-but-not-sufficient for good-quality evidence.

Broad Use¶

Clinical trials and biomedicine (canonical regulatory context): Pre-specified power-based sample-size calculation is required for most regulatory submissions. FDA Type B meetings discuss design including power assumptions. ICH E9 (Statistical Principles for Clinical Trials) addresses power, with conventional 80% or 90% power for primary endpoints. The minimum clinically important difference (MCID)—the effect size below which treatment differences are not clinically meaningful—provides the effect-size assumption for power analysis^[5]. Adaptive-trial designs may modify sample size based on interim-analysis effect-size estimates (sample-size re-estimation), preserving overall power while reducing expected sample size. Non-inferiority and equivalence trials have their own power considerations, with the equivalence margin rather than zero as the target detection threshold.

Psychology and behavioral science (canonical reform context): Cohen's 1962 critique documented that much psychological research was underpowered (typical power of 0.18 for small effects, 0.48 for medium). Despite Cohen's tools and advocacy, power in psychology remained low through the late 20^th century. The 2010s replication crisis brought renewed attention to underpowered research; meta-analyses showed median powers of 20-35% in many psychology subfields. Reform including pre-registered power analyses, power-justified sample sizes, and journal policies requiring minimum power or sample-size justification.

Physics and particle physics: Signal-detection power at the 5σ discovery threshold requires understanding of background rates, signal rate assumptions, and accumulated luminosity. LHC experiment design considered power at various Higgs-mass hypotheses. Dark-matter direct-detection experiment design computes power to detect signals of various cross-sections. Gravitational-wave detector sensitivity characterized through power to detect various waveform types at various distances.

Epidemiology and public health: Cohort-study power for detecting moderate risk ratios given disease prevalence and exposure distribution; case-control study power with fixed number of cases. Cluster-randomized trial power with design-effect adjustment for intra-cluster correlation. Vaccine-efficacy trial power—COVID-19 mRNA vaccine Phase 3 trials were explicitly powered on event rates to detect specified efficacy thresholds within event-driven trial designs. Seroprevalence surveys powered to estimate prevalence with specified precision.

Technology and A/B testing: Sample-size calculators are standard in experimentation platforms (Optimizely, VWO, Google Optimize, in-house tools). Minimum-detectable-effect reporting standard in test reports. Group-sequential designs allowing early stopping for overwhelming benefit or futility through power-adjusted analyses^[5]. Multi-armed bandit alternatives that allocate traffic based on accumulating evidence rather than pre-specified power targets.

Genetics and genomics: GWAS power at genome-wide significance (5×10⁻⁸) requires enormous samples for detecting common-variant effects (often hundreds of thousands of individuals through consortia like UK Biobank, 23andMe research cohorts, Million Veteran Program). Rare-variant association tests have distinct power considerations (burden tests, SKAT). eQTL and other molecular-QTL studies powered for detecting expression effects.

Quality control: Acceptance-sampling plan design specifying α (producer's risk) and β (consumer's risk) with operating-characteristic curves showing power as a function of true defect rate. Process-capability study design with specified power to detect meaningful process-mean or process-variance deviations.

Agricultural experiments: Randomized-complete-block and factorial-design power calculations accounting for block variance and interaction structure. Long-term rotation-experiment design for detecting cumulative effects.

Ecology and conservation biology: Power to detect population declines of specified magnitude over monitoring periods; marine-protected-area effect-detection power; biodiversity-change detection in long-term monitoring. These often involve small sample sizes (limited sites), high variability (natural-system noise), and multi-decadal time horizons—power challenges are severe.

Educational research: Cluster-randomized school-trial power considering school-level intraclass correlation, typically requires 40-80 schools per arm for detectable effect sizes in the 0.10-0.20 standardized-effect range. What Works Clearinghouse standards specify minimum power thresholds for evidence classification.

Clarity¶

The power analysis frame makes explicit the specific quantity—probability of rejecting H₀ under a specified H₁—and makes explicit the four-way relationship among power, effect size, sample size, and α. Without the frame, people run studies whose sample sizes are determined by convenience or convention without regard for what effects they can detect, interpret non-significant results as evidence of no effect regardless of achieved power, and conflate "observed power" with independent diagnostic information^[6]. With the frame, diagnosis becomes specific: What is the minimum effect size of scientific interest, and on what basis? What variability is expected in the outcome, and what data inform that assumption? What significance level is appropriate, and what power target? What sample size achieves the required power? Can design efficiency (blocking, paired design, covariate adjustment, paired-alternative designs) reduce the required n? How sensitive is the analysis to the effect-size assumption—what is the minimum detectable effect at the planned sample size? Is the study adequately powered for the primary and any key secondary analyses? If not, should the study be redesigned, or should findings be interpreted with explicit acknowledgment of power limitations? The frame clarifies that power is a study-design and interpretation parameter, not just a sample-size formula.

Manages Complexity¶

Decomposes study design into structured components (effect size, variability, α, n, power) with explicit relationships, supporting principled trade-off analysis. Cross-domain transfer is productive: Cohen's power-analysis framework from psychology to education to medicine; design-effect-adjusted power from public-health trials to educational cluster trials to technology-platform A/B testing; sequential-design power from clinical trials to quality control to online experimentation; simulation-based power from complex-model statistics to ML evaluation^[2]. The decomposition reveals interplay with other primes: hypothesis testing (#434)—power is a property of the test framework; type I / type II errors (#445)—power is 1−β, tightly paired; effect size (#447)—power calculation requires a specified effect size, tightly paired; statistical significance / p-value (#435)—power and α jointly shape the test's operating characteristics; confidence intervals (#436)—CI width is the estimation-analog of power, and sample-size calculation can target CI width instead of power; sampling representativeness (#433)—design effect scales power; reproducibility (#441)—underpowered studies contribute to replication failures through winner's curse.

Abstract Reasoning¶

The analyst asks: What is the scientific question, and what is the minimum effect size that would matter scientifically, clinically, or practically? What prior evidence informs expected effect size and variability—from pilot studies, meta-analyses, domain knowledge? What significance level α is appropriate, and what power target—80% conventional, 90% for high-stakes, something else? What sample size is required given these inputs, and is that feasible—can we collect this n, or do we need to redesign to reduce variability, use paired designs, or relax the expected minimum effect? How sensitive is the required n to the effect-size assumption—small variations in expected δ can produce large variations in required n for small effects^[3]? What design efficiencies can reduce required n—blocking, paired alternatives, covariate adjustment, matched designs? If sample size is fixed by constraints, what is the minimum detectable effect—is the study adequately powered for any scientifically meaningful effect, or is it pre-determined to be underpowered? For cluster or hierarchical designs, what is the design effect, and how does it scale required n? If running multiple tests, does multiplicity adjustment further increase required n? Mature practice conducts pre-study power analysis with domain-justified effect-size assumptions, reports sample-size justification in study protocols and publications, considers sensitivity to assumptions, and interprets results in light of achieved power. Immature practice skips power analysis, uses generic Cohen conventional effect sizes without domain justification, runs convenience-size studies, and reports "post-hoc power" as if it provided diagnostic information.

Knowledge Transfer¶

Domain	Typical effect-size assumption	Typical power target	Characteristic difficulty
Clinical trial superiority	Minimum clinically important difference (MCID)	80-90%	MCID justification from prior evidence
Psychology experiment	Cohen's d = 0.5 (medium) or domain-specific	80%	Effect-size inflation; underpowering
GWAS	0.5-2% per-variant variance explained	80% at 5×10⁻⁸	Requires ≥10⁴-10⁵ samples
Cluster-randomized trial	0.10-0.30 standardized ICC-adjusted	80%	Design-effect doubling or tripling required n
A/B test (tech)	1-5% relative lift on metric	80%	Metric variance; sample-size vs. test-duration trade
Cohort study	Relative risk 1.3-2.0	80%	Prevalence-driven expected-event count
Case-control study	Odds ratio 1.5-3.0	80%	Case accrual; matched design gains
Quality control acceptance	Lot defect rate vs. AQL	Producer's/consumer's risk	OC-curve shape across lot rates
Ecology monitoring	10-30% population change	80% over N years	High natural variability; long horizons
Educational trial	0.15-0.25 standardized	80%	Cluster structure; implementation variation

Examples¶

Formal/abstract¶

The Pfizer-BioNTech BNT162b2 Phase 3 COVID-19 vaccine trial (Polack et al. 2020, NEJM) is a high-profile example of power-based trial design in a high-stakes regulatory context. The trial was an event-driven Phase 3 randomized placebo-controlled trial with approximately 44,000 participants randomized 1:1 to vaccine or placebo. The primary endpoint was confirmed COVID-19 occurring at least 7 days after the second dose of vaccine. The power calculation was event-driven rather than sample-size-driven: the trial was designed to continue until a pre-specified number of COVID-19 cases accumulated across the combined trial arms, at which point the efficacy analysis would be conducted. The pre-specified success criterion for the primary efficacy analysis was that the lower bound of the 95% confidence interval for vaccine efficacy (1 minus the hazard ratio) exceed 30%—a pre-specified regulatory threshold set by the FDA for COVID-19 vaccines in October 2020^[5]. Power calculation: with 164 cases across arms, the trial had approximately 90% power to rule out vaccine efficacy ≤ 30% if true efficacy was ≥ 60%; with pre-specified interim analyses at 32, 62, 92, and 120 cases, the trial could demonstrate success at interim if effects were large. The trial's final analysis was conducted at 170 cases (162 placebo, 8 vaccine), yielding an estimated vaccine efficacy of 95.0% (95% CI 90.3-97.6%), with the CI lower bound vastly exceeding the pre-specified 30% threshold. The event-driven power design illustrates several features of modern power analysis in clinical trials: (i) Event-driven rather than fixed-sample-size allows the trial to accumulate the number of events required for the efficacy analysis, regardless of how long it takes—particularly important during a rapidly-evolving pandemic where incidence rates were uncertain. (ii) Sequential-analysis power with pre-specified looks at 32, 62, 92, 120, and 164 cases, using O'Brien-Fleming-style α-spending to preserve overall Type I error near 2.5% one-sided while allowing early success at interim if effects were large. (iii) Regulatory-threshold design — the FDA's October 2020 guidance required demonstrated vaccine efficacy > 50% with lower 95% CI bound > 30%, shaping the power calculation around that threshold rather than around rejecting the null of zero efficacy^[5]. (iv) Large-effect sensitivity — the trial was substantially overpowered for the observed effect size (95% efficacy vastly exceeded the 60% assumption), allowing early interim success; the design was appropriately conservative given regulatory stakes. (v) Post-approval effectiveness — Phase 3 efficacy estimates represent a specific population (adults meeting inclusion criteria during trial period); real-world effectiveness studies extended evidence to broader populations. The BNT162b2 trial exemplifies power analysis as practical application of the Neyman-Pearson framework at the intersection of regulatory decision-making, pandemic response, and accelerated clinical development.

Mapped back: This case illustrates the structural signature of power analysis—pre-specification of minimum detectable effect (vaccine efficacy ≥ 60% vs 30% regulatory threshold), event-driven sample-size determination, sequential-analysis α-spending for multiple looks, regulatory-threshold calibration—and the core abstraction that "adequate power depends jointly on α, effect size, variability, and sample size"; the event-driven design shows how power analysis applies even when sample size is random (driven by event accumulation) so long as the relationship between power and the components is pre-specified.

Applied/industry¶

A regional agricultural extension office is designing a multi-farm study to evaluate a new cover-cropping system intended to reduce nitrogen fertilizer requirements while maintaining yields on corn-soybean rotations in the regional climate. The cover-crop system has been tested in small-plot research-farm settings showing yield maintenance at reduced fertilizer rates, but on-farm implementation across varied soil types and management intensities is needed before regional recommendations. The extension economics and agronomy teams commission a randomized on-farm trial with the following power-driven design: (a) Primary outcome: Corn yield (bushels per acre) in the cover-crop system versus conventional fertilizer management, averaged across two years of the rotation. (b) Minimum detectable difference of interest: 5 bushels per acre on a regional baseline of approximately 190 bushels per acre (a 2.6% difference)—the extension economists judge that smaller yield differences are indistinguishable from management-variation noise and larger differences would materially affect grower adoption economics^[4]. © Expected variability: Between-farm standard deviation of approximately 15 bushels per acre based on historical county-level yield variation adjusted for management; within-farm (split-field) variability approximately 8 bushels per acre based on small-plot research-farm data. (d) Design choice—split-field versus between-farm: Split-field design (half of each participating farm's experimental acres under cover-crop, half conventional, randomized within farm) reduces effective variability from the between-farm SD to the within-farm SD; this design-efficiency choice dramatically reduces required sample size. (e) Power calculation: Paired-design power with 8 BPA within-farm SD, 5 BPA target effect, α = 0.05 two-sided, 80% power target—requires approximately 24 participating farms for the 2-year rotation. (f) Sample-size feasibility: The extension office has commitments from 32 farms across the region, providing margin for potential dropouts and mixed-farm-size adjustments. (g) Sensitivity analysis: The team examines required n under variability assumptions of 10 BPA (more variable, requiring n ≈ 38 farms) and 6 BPA (less variable, requiring n ≈ 14 farms)—the 32-farm commitment covers the expected and higher-variability cases. (h) Secondary-endpoint power: Fertilizer reduction (clearly detectable; power near 100% at expected effect), soil-health indicators (less clearly powered at 32 farms; noted as exploratory with power caveats), and net economic return per acre (adequate power given yield-and-fertilizer primary estimates)—all documented in the analysis plan. (i) Analysis methods: Mixed-effects model with farm-level random effects and year fixed effects, with cluster-robust standard errors at the farm level; pre-specified sensitivity analysis using non-parametric paired tests if distributional assumptions are violated. Over the 2-year study period, the trial proceeds with 29 farms completing both years of data collection (three farms dropped due to management changes or data-quality issues—within the expected-dropout margin). Analysis: Estimated yield difference is −1.2 bushels per acre (cover-crop minus conventional), 95% CI (−4.1, +1.7) BPA. The CI includes zero and is centered near zero; the 5-BPA clinical-threshold for meaningful yield penalty is outside the CI upper bound (+1.7 < 5.0)^[7]. Interpretation: The data are consistent with no meaningful yield penalty (point estimate within 1.2 BPA of equivalence; CI upper bound well within the 5-BPA threshold of concern); the cover-crop system appears yield-neutral at the reduced fertilizer rate within the power of the study. The team's pre-specified analysis also estimates substantial fertilizer reduction (42 lbs/acre less N applied, CI tight and well-separated from zero) and positive net economic return per acre (point estimate $18.50/acre higher, CI from $4 to $33). The regional recommendation resulting from the study: Cover-crop system is recommended as yield-neutral fertilizer-reduction option for corn-soybean rotations in the region, with secondary benefits (soil health indicators, erosion reduction) consistent with research-farm findings but recognized as less fully powered. The case illustrates power analysis as practical application to agricultural field-trial design: minimum-detectable-difference specification rooted in extension-economics judgment, within-farm variability estimation from pilot data, design-efficiency choice (split-field paired) reducing required n, sensitivity analysis across variability assumptions, feasibility check against recruitment, secondary-endpoint power acknowledgment, and pre-specified analysis plan integrating effect-size estimation with power-based trial design.

Mapped back: This case exemplifies the structural signature of power analysis—minimum-detectable-difference (5 BPA) specification rooted in practical decision stakes, within-design-block variability estimation, design-efficiency gains from split-field pairing reducing required farm-count from ~38 to ~24, sensitivity analysis across assumed variability ranges—and the core principle that power analysis bridges experiment planning and decision-making: the pre-specified 5-BPA threshold operationalizes "what would matter," the 80% power target reflects the farmers' tolerance for missing real effects, and the final CI (upper bound 1.7 BPA) below the 5-BPA threshold allows confident conclusion of yield-neutrality.

Structural Tensions¶

T1 — Adequate-power standards versus resource and feasibility constraints. Detecting small effects with high power requires large samples that can be infeasible for many research contexts (rare diseases, field experiments with limited sites, early-phase research with limited resources). The standards-feasibility tension leads to either under-powered studies that publish inflated estimates and contribute to replication failures, or to adequately-powered studies that pool across sites/time through consortia or meta-analyses. Mature practice acknowledges the sample-size-for-effect-size reality, combines data across studies through consortia, designs for realistic rather than aspirational power, and is explicit about power limitations in interpretation.

T2 — Specified-effect-size versus unknown-effect-size power analysis. Power analysis requires a specified effect size under H₁, but the "true" effect size is unknown—that is why the study is being conducted. Effect-size specification can come from (a) minimum clinically/practically important difference judgments, (b) prior pilot-study or meta-analytic estimates, © Cohen's conventional small/medium/large benchmarks, or (d) subjective-but-transparent guesses. Each source has limitations—MCIDs may not exist in novel research areas; pilot estimates suffer from small-sample noise and publication bias; Cohen's conventions are domain-indifferent and may not fit; guesses lack grounding. Mature practice combines sources, reports sensitivity across plausible effect sizes, and justifies the chosen target; immature practice picks a convention without justification or uses overly optimistic effect sizes to reach feasible sample sizes^[3].

T3 — Planning-phase power versus execution-phase observed power. Pre-study power analysis informs design; post-hoc "observed power" computed from results is mechanically linked to the p-value and provides no independent diagnostic. The temptation to compute observed power after non-significant results is widespread but methodologically problematic. Legitimate post-study analysis uses pre-specified or external effect-size estimates to characterize what the study could have detected, or quantifies the minimum effect size the study had adequate power to detect (sensitivity post-hoc power analysis, Lakens 2017). Mature practice conducts a priori power analysis, avoids post-hoc observed power, and uses external or pre-specified effect-size references for sensitivity retrospection; immature practice treats post-hoc observed power as a legitimate diagnostic tool.

T4 — Type I versus Type II error rate balance calibrated to decision context. The conventional α = 0.05 and β = 0.20 (power = 80%) implies a 4:1 ratio of Type I to Type II error rates—a Fisher-era judgment that false positives are more costly than false negatives. Many decision contexts have different cost structures: (a) Safety monitoring—false negative (missing a safety signal) may be far more costly than false positive, warranting lower β. (b) Screening contexts—false positive may trigger costly confirmatory testing and patient anxiety, warranting lower α. © Exploratory research—moderate α and β may both be acceptable if the goal is hypothesis generation. (d) Regulatory approval—asymmetric costs may justify specific α and β choices. The default 4:1 ratio is unreflective; decision-theoretic power analysis calibrates α and β to the specific decision context. Mature practice chooses α and β based on decision stakes and prior probabilities; immature practice defaults to α = 0.05, β = 0.20 without question.

T5 — Parametric assumptions versus robust nonparametric alternatives. Power calculations for parametric tests (t, F) assume normality and homogeneity of variance. Real-world data often violate these assumptions; nonparametric tests (Mann-Whitney, Kruskal-Wallis, permutation tests) are more robust but have lower power under the parametric assumptions and higher power under assumption violations. Power calculations for nonparametric tests are complex and often require simulation. The tension is between the convenience of parametric power formulas and the robustness of nonparametric methods. Mature practice conducts sensitivity analyses with both parametric and nonparametric power estimates; immature practice assumes parametric power formulas apply without checking assumptions.

T6 — Single-study power target versus cumulative evidence meta-perspective. Traditional power analysis targets 80% power for a single study. But individual underpowered studies can be combined through meta-analysis to achieve stronger cumulative evidence. The tension is between designing each study adequately powered (expensive, sometimes infeasible) and designing many smaller studies for meta-analytic synthesis (requires protocol pre-specification and publication-bias management). Mature consortial science (UK Biobank, GENIE cancer genomics, mega-analytic consortia in psychology and social science) designs studies with realistic power targets and commits to meta-analytic synthesis; immature practice targets isolated 80%-powered studies that may not replicate.

Structural–Framed Character¶

Statistical Power sits at the structural end of the structural–framed spectrum: it is a pure relational pattern, the same in any domain where it appears, and nothing about its meaning depends on a particular field's vocabulary or assumptions.

The prime is a mathematical relationship: the probability that a test correctly detects a real effect, bound rigidly to effect size, sample size, and the chosen error threshold, so that fixing any three pins down the fourth. It carries no evaluative charge and presupposes no human institution; it is a property of a detection procedure, and the same arithmetic governs a clinical trial, a quality-control screen, or a sensor's ability to flag a signal against noise. Using it means computing a quantity that is already implied by the test's structure, not importing an outside perspective. On every diagnostic, it reads structural.

Substrate Independence¶

Statistical Power is a narrowly substrate-independent prime — composite 2 / 5 on the substrate-independence scale. It is tightly tied to hypothesis-testing design within formal statistics, and although its underlying idea — sensitivity to detecting a true effect — has conceptual cousins in engineering and detection systems, the construct itself is operationalized almost entirely within the causal-inference and experimental-design ecosystem. Any move beyond that is largely metaphorical, and a practitioner would not reach for power outside a statistical setting without significant translation. The signal-detection intuition gives it a foothold, but the prime stays tethered to its statistical home.

Composite substrate independence — 2 / 5
Domain breadth — 2 / 5
Structural abstraction — 3 / 5
Transfer evidence — 2 / 5

Relationships to Other Primes¶

Parents (2) — more general patterns this builds on

Statistical Power presupposes Experimental Design

Statistical power presupposes experimental design because the quantities binding power — effect size, sample size, significance level, variability — are all set by the design choices that allocate units to treatments, specify outcome measurement, and control noise. It inherits experimental design's commitment to principled architecture for causal inference under constraints, and operates as the diagnostic that asks whether the design is adequately sized to detect a true effect. Power analysis is design's planning tool.
Statistical Power presupposes Probability

Statistical power presupposes probability because power = P(reject H0 | H1 true) is itself a probability assignment over decision outcomes governed by Kolmogorov's coherence rules. The four quantities binding power -- effect size, sample size, significance level, variability -- enter through the sampling distribution of the test statistic, which is a probability object. Without probability's apparatus for combining, conditioning, and normalizing degrees of belief or frequency, the Type I and Type II error rates and their trade-off have no formal home; power IS conditional probability deployed in a decision frame.

Path to root: Statistical Power → Probability

Neighborhood in Abstraction Space¶

Statistical Power sits in a sparse region of abstraction space (98^th percentile for distinctiveness): few abstractions share its structure, so a faithful description tends to retrieve it precisely rather than landing on a neighbor.

Family — Frequentist Hypothesis Testing (3 primes)

Nearest neighbors

Statistical Significance (p-Value) — 0.79
Hypothesis Testing (Null vs. Alternative) — 0.77
Multiple Comparisons Correction — 0.72
Type I & Type II Errors — 0.71
Sampling (Representativeness) — 0.71

Computed from structural-signature embeddings · 2026-05-29

Not to Be Confused With¶

Statistical Power must be distinguished from Statistical Significance (p-value), its nearest neighbor in the hypothesis-testing framework. Both are properties of statistical tests, but they describe different aspects of test performance. Statistical Power answers the forward-looking question: "Given that a true effect exists at a specified magnitude, what is the probability that my test will detect it?" Statistical Significance (p-value), by contrast, answers the backward-looking question: "Given that the null hypothesis is true (no effect), what is the probability of observing data as extreme as what I observed?" These are the complement of two different error types in the Neyman-Pearson framework: power is 1 − β (avoiding Type II error, false negatives); significance level α controls Type I error (false positives). A test can have high power yet low significance (a sensitive test that rarely rejects the null when it is true, but readily detects true effects), or low power yet low significance (a conservative test that rarely declares significance either way). The confusion between them is pervasive: practitioners sometimes report "observed power" computed from data as if it were a diagnostic of test sensitivity, when in fact observed power is a deterministic function of the p-value and provides no independent information. Correct practice conducts power analysis before the study to design adequate sample size, then interprets the p-value after the study in light of the pre-specified power and effect-size assumptions.

Nor is Statistical Power identical to Statistical Inference, the broader epistemic framework for drawing conclusions from samples to populations. Statistical Inference encompasses hypothesis testing (including power considerations), parameter estimation with uncertainty quantification, causal inference methods, and model evaluation—a much wider epistemic scope than power alone. Power is a tool within the statistical inference toolkit, specifically for designing tests that have adequate sensitivity to detect effects of scientific or practical importance. A study can have adequate power yet produce inferences that are misleading due to confounding, selection bias, or measurement error—power addresses only the sensitivity of the test to effect magnitude, not the validity of causal claims or the absence of other threats to inference. Conversely, a study with lower power may still yield valuable inferences about effect magnitude through confidence intervals and effect-size estimation, even if hypothesis testing power is limited. The distinction matters because a researcher might focus narrowly on achieving 80% power (a narrow test-design criterion) without attending to other components of sound inference—representative sampling, absence of confounding, measurement reliability, appropriate statistical modeling. Power is necessary but not sufficient for valid statistical inference.

Statistical Power is also distinct from Effect Size, though the two are tightly paired and often discussed together. Effect size is the magnitude of an effect—the true difference between groups, the strength of association, the practical meaningfulness of a phenomenon—measured independently of sample size. Statistical Power is the probability of detecting a specified effect size given a particular sample size and significance level. Power depends on effect size (larger effects are easier to detect, requiring smaller sample sizes for the same power; smaller effects require larger samples), but they are not the same quantity. A study can have a large effect size (large true difference) yet low power if the sample size is too small relative to the variability. Conversely, a study can have high power to detect small, statistically significant differences that are practically meaningless. The relationship is captured in the power-analysis formula: power is a function of the four quantities (n, α, δ, σ), and any three of them determine the fourth. Specifying an effect size is a prerequisite for power calculation, not a consequence of it. The tension is that effect size must be chosen before the study is conducted (via MCID specifications, prior research, domain knowledge), but the "true" effect is unknown, making effect-size specification inherently uncertain. Practitioners sometimes choose effect-size assumptions to make sample-size requirements feasible (computing backward from budget to effect size) rather than specifying effect size based on scientific judgment, a practice that undermines the validity of power-based design.

Solution Archetypes¶

Solution archetypes in the catalog that build on this prime — directly (this prime is a source ingredient) or as a related prime.

Built directly on this prime (1)

Hypothesis Test Power Calibration

Also a related prime in 15 archetypes

▸ Show 5 more

Notes¶

Experimental-design/statistics origin (Neyman-Pearson 1933 formalized power as complement of Type II error; Cohen 1962, 1969, 1988 established power analysis as systematic practice in social sciences). The tight_pair_with_type_i_type_ii_errors flag reflects that power is 1−β, constitutive of the Neyman-Pearson framework; reciprocal flag should be wired into #445. The tight_pair_with_effect_size flag reflects that power analysis cannot be conducted without a specified effect size—the two primes are mutually definitional; reciprocal flag should be wired into #447. Related primes: #434 hypothesis_testing_null_vs_alternative (framework within which power is defined), #445 type_i_type_ii_errors (tight pair), #447 effect_size (tight pair), #435 statistical_significance_p_value (α and power jointly shape operating characteristics), #436 confidence_intervals (CI-precision-based sample-size planning as alternative to power-based), #433 sampling_representativeness (design-effect scales power), #441 reproducibility_replicability (underpowered studies contribute to replication failures through winner's curse), #432 randomization (randomization-based tests' power). Strong transfer targets: clinical-trial and vaccine-trial regulatory design, psychology-replication sample-size justification, GWAS multi-cohort meta-analysis, cluster-randomized-trial design across public health and education, A/B-testing platform sample-size calculators, agricultural and ecological trial design. Pass B should develop archetypes for a-priori-sample-size-planning, sensitivity-to-effect-size power analysis, design-efficiency-through-paired/blocked-designs, sequential-and-adaptive sample-size re-estimation, cluster-design design-effect adjustment, simulation-based power for complex designs, event-driven trial power (vaccine trials, time-to-event), and power-based interpretation of non-significant results.

References¶

[1] Neyman, J., & Pearson, E. S. (1933). On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society A, 231(694–706), 289–337. Foundational paper: frames inferential conclusions as tentative decisions with controlled long-run error rates, subject to revision as new data accumulate. ↩

[2] Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences (2^nd ed.). Lawrence Erlbaum Associates. Foundational text on power analysis: links sample size, effect size, significance threshold, and noise level into a coherent design discipline — the practical instantiation of "set decision thresholds appropriate to the noise level" for empirical research. ↩

[3] Lakens, D. (2022). Sample size justification. PsyArXiv Preprints. Lakens comprehensive guide to justified sample-size specification beyond traditional power analysis. ↩

[4] Cox, D. R. (1958). Planning of Experiments. John Wiley & Sons. Canonical exposition of how active intervention—assigning units to treatments and pre-specifying measurement—isolates causal effects from confounding across scientific domains. ↩

[5] Polack, F. P., et al. [BNT162b2 Trial Group]. (2020). Safety and efficacy of the BNT162b2 mRNA Covid-19 vaccine. New England Journal of Medicine, 383(27), 2603–2615. Pfizer-BioNTech COVID-19 vaccine trial with event-driven power-based design and sequential analysis. ↩

[6] Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2016). Statistical tests, P values, confidence intervals, and power: A guide to misinterpretations. European Journal of Epidemiology, 31(4), 337–350. Authoritative critique of statistical practice: exposes how implicit distributional assumptions and convenience-driven model choices generate misinterpretations of significance and uncertainty. ↩

[7] Cumming, G. (2014). The new statistics: why and how. Psychological Science, 25(1), 7–29. Cumming new statistics effect-size confidence intervals point estimate plus uncertainty reporting discipline. ↩

[8] Cohen, J. (1962). The statistical power of abnormal-social psychological research: A review. Journal of Abnormal and Social Psychology, 65(3), 145–153. Cohen's early critique of underpowering in psychology research triggering power-analysis development.

[9] Gelman, A., & Carlin, B. (2014). Beyond power calculations: Assessing type S (sign) and type M (magnitude) errors. Perspectives on Psychological Science, 9(6), 641–651. Gelman-Carlin extending error-rate concepts to effect-size estimation sign and magnitude errors.

[10] Sedlmeier, P., & Gigerenzer, G. (1989). Do studies of statistical power have an effect on the power of studies? Psychological Bulletin, 105(2), 309–316. Sedlmeier-Gigerenzer documenting persistent underpowering in psychology despite power-analysis availability.

[11] Ioannidis, J. P. A. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124. Foundational analysis of how publication bias, low statistical power, and flexible analytic choices produce a literature in which most positive findings fail to replicate—motivating epistemic humility about scientific claims.

[12] Benjamin, D. J., et al. (2018). Redefine statistical significance. Nature Human Behaviour, 2(1), 6–10. Benjamin et al. advocating for α = 0.005 discovery threshold to address power and prior-probability issues.

[13] Wilkinson, L., & American Psychological Association Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: guidelines and explanations. American Psychologist, 54(8), 594–604. Wilkinson APA task force statistical methods effect-size reporting confidence intervals significance testing.

[14] Lehmann, E. L. (1993). The Fisher, Neyman-Pearson theories of testing hypotheses: One theory or two? Journal of the American Statistical Association, 88(424), 1242–1249. Lehmann canonical historical treatment of Fisher-Neyman-Pearson philosophical and methodological differences.

[15] Student. (1908). The probable error of a mean. Biometrika, 6(1), 1–25. Gosset t-distribution foundational for small-sample error-rate control in hypothesis testing.