Simulation-Based Power Analysis¶
Monte Carlo simulation — instantiates Hypothesis Test Power Calibration
Estimates power for a complex or nonstandard design by repeatedly generating synthetic datasets under an assumed effect and running the actual planned analysis on each.
The single idea that makes this mechanism itself and not its analytic twin: it estimates power by building the data. Where a formula exists only for a handful of clean designs, most real studies are messier — units clustered inside sites, repeated measurements on the same subject, interim looks, non-normal outcomes, an estimator with no tidy variance expression. Simulation-Based Power Analysis sidesteps the missing formula by specifying a full data-generating process that encodes the effect you want to detect and every complication that erodes sensitivity, drawing thousands of synthetic datasets from it, running the exact analysis you plan to run on each, and counting the fraction that reject the null. That fraction is the power estimate. Because the whole pipeline — noise structure, missingness, clustering, the actual test — runs on every replicate, the mechanism can price sensitivity for designs no closed form can touch. Its defining commitment is fidelity: it does not approximate the design, it enacts it.
Example¶
A district is planning a cluster-randomized trial of a new reading curriculum: whole classrooms, not individual pupils, are assigned to the new program or business-as-usual, and the outcome is an end-of-year comprehension score. No clean two-group formula fits, because pupils in the same classroom resemble each other — an intraclass correlation of roughly 0.15 in the district's records — and every extra correlated pupil adds less independent information than a formula assuming independence would credit. So the team writes a generator: draw classroom-level effects, draw pupil scores around them with the assumed correlation, shift the treated classrooms by the target gain (an illustrative third of a standard deviation), then delete about 12% of pupils to mimic mid-year moves, and finally fit the mixed-effects model they actually intend to use. Run that a few thousand times and count rejections. The first pass reports power near 62% for 40 classrooms — well short of target — and, crucially, shows why: the loss is dominated by the clustering, not the dropout. That points the redesign straight at recruiting more classrooms rather than fewer pupils each.
How it works¶
- Specify the generative model, not a formula. Write down how outcomes arise: fixed effect of the meaningful size, random effects for clusters, residual noise, correlation structure — the honest data-generating process.
- Inject the complications. Attrition, intermittent missingness, non-normal tails, compliance dilution, and interim-look rules are written into the generator so they hit power the way they will in the field.
- Replicate and re-analyze. For each of many synthetic datasets, apply the planned analysis verbatim — the same model, the same corrections, the same stopping rule.
- Count rejections as power. The share of replicates that reject the null under the assumed true effect estimates power; the Monte Carlo noise in that share shrinks with more replicates.
Tuning parameters¶
- Number of replicates — more runs tighten the power estimate's own sampling error but cost compute; too few and the estimate wobbles run to run.
- Data-generating fidelity — how faithfully the generator mirrors reality (tails, correlation, missingness mechanism); higher fidelity is more trustworthy but harder to justify.
- Assumed true effect — the effect size baked into the generator; power is reported at that value, so it must be the meaningful one, not a hopeful one.
- Missingness mechanism — completely-at-random versus outcome-dependent dropout changes power sharply and must be chosen deliberately.
- Analysis lock — whether the simulated analysis exactly matches the intended one; any drift makes the estimate answer a different design.
When it helps, and when it misleads¶
Its strength is reach: it prices power for clustered, longitudinal, adaptive, or nonstandard designs that formulas cannot express, and it exposes which complication dominates the loss. It misleads through its own inputs — a simulation is only as honest as the generator, so a rosy variance or an at-random missingness assumption yields a confident number for a design that will not behave that way in the field. A subtler trap is treating the replicate count as free: too few replicates and the reported power carries a Monte Carlo standard error large enough that the same design can read "adequately powered" or "underpowered" on consecutive runs.[n1] The guarding discipline is to report the simulation's own uncertainty, stress the generative assumptions rather than fixing them at their most flattering, and keep the simulated analysis byte-for-byte identical to the planned one.
How it implements the components¶
operating_characteristic_model— supplies the simulation flavor of the model: rejection probability estimated by Monte Carlo rather than by formula.noise_and_variance_profile— the generator explicitly encodes residual variance, clustering, and correlation, so the noise structure is enacted, not assumed away.attrition_and_missingness_allowance— dropout and missing observations are written into the data-generating process and hit power as they would in practice.null_alternative_and_estimand_frame— the effect, estimand, and full analysis (including nonstandard estimators and stopping rules) are instantiated in code, fixing exactly what is being tested.
It does not build a sensitivity_scenario_grid — one simulation prices one data-generating process, not a matrix of rival assumptions; sweeping those belongs to Power Sensitivity Grid. Nor does it set a pre_registered_interpretation_rule; committing the reading of the result in advance is Pre-Analysis Power Statement. Its nearest twin is Closed-Form Power Calculation: the two share the operating-characteristic role, but simulation alone enacts the noise-and-attrition structure (its own machinery) while the formula reaches the answer deterministically, without replication.
Related¶
- Instantiates: Hypothesis Test Power Calibration — the engine for designs beyond the reach of formulas.
- Consumes: Pilot Variance Estimation — supplies the variance, correlation, and dropout inputs the generator needs.
- Sibling mechanisms: Closed-Form Power Calculation · Minimum Detectable Effect Table · Operating Characteristic Curve · Power Sensitivity Grid · Pre-Analysis Power Statement · Pilot Variance Estimation
Editorial Notes¶
Form Classification¶
Form family: Analysis, Modeling & Optimization
Rationale: Simulation-Based Power Analysis operates as an analytical, modeling, inference, comparison, or optimization procedure that derives insight or a solution because it estimates power for a complex or nonstandard design by repeatedly generating synthetic datasets under an assumed effect and running the actual planned analysis on each.
Independent corroboration: The frozen evidence defines Simulation-Based Power Analysis as 'Estimates power for a complex or nonstandard design by repeatedly generating synthetic datasets under an assumed effect and running the actual planned analysis on each', so its operative form is Analysis, Modeling & Optimization.
Nearest alternative: Experiment, Test & Rehearsal — Simulation-Based Power Analysis includes features of an active test, trial, simulation, drill, or rehearsal that generates evidence through a deliberate attempt or perturbation, but its defining operation is an analytical, modeling, inference, comparison, or optimization procedure that derives insight or a solution.
Review outcome: Independent reviewer agreement; medium confidence.
Origin Attribution¶
Primary origin: Statistics & Experimental Design
Origin pattern: Single lineage
Present-day reach: Multi-domain
Rationale: Repeatedly generating data under an assumed effect and rerunning the planned analysis is Monte Carlo statistical power analysis.
Related originating lineages:
- Computer Science & Software Engineering — Efficient computation makes large repeated analyses feasible.
- Data Science & Analytics — Complex pipelines and nonstandard estimators often require empirical simulation.
- Mathematics — Mathematical modeling, proof, and abstract-structure practice supplies a parallel or contributing lineage for the mechanism's defining operation: estimates power for a complex or nonstandard design by repeatedly generating synthetic datasets under an assumed effect and running the actual planned analysis on each.
- Medicine & Healthcare — Clinical and clustered study designs are major applications of simulation-based sizing.
Review resolution: The blind reviewers agree that statistics_experimental_design is the primary origin and differ only on alternate origin disagreement, domain reach disagreement. I preserve every independently explained alternate from both records rather than imposing a numeric cap. I retain single_lineage because the combined evidence shows one traceable formative lineage. The broader reach of multi_domain records portability separately from historical provenance; encyclopedia_synthesis=false preserves the affirmative synthesis judgment where either reviewer identified one.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
[n1] A simulated power estimate is itself a proportion measured over finite replicates, so it carries a Monte Carlo standard error that shrinks only as the number of runs grows. Reporting a power figure without that band invites over-reading a difference that is just simulation noise — the same design can flip across the target line between two runs of a few hundred replicates. ↩