Power Sensitivity Grid¶
Robustness grid — instantiates Hypothesis Test Power Calibration
Recomputes power across a grid of alternative variance, attrition, and compliance assumptions to expose designs that only clear the bar under optimistic inputs.
The single idea that makes this mechanism itself and not the effect-sweeping curve: it varies the nuisance assumptions, not the effect. A power calculation rests on inputs nobody knows exactly — the true variance, the dropout rate, how many assigned participants actually comply, how correlated clustered units are. A single point estimate for each produces a single power figure that looks decisive but silently depends on every one of those guesses being right. Power Sensitivity Grid holds the meaningful effect fixed and re-runs the power calculation across a matrix of plausible values for the nuisance parameters — a cell for each combination of assumed variance, attrition, and compliance — turning one number into a landscape. Where the landscape stays above target across the whole grid, the design is robust. Where power collapses in a corner of pessimistic-but-credible assumptions, the grid has found the design's fragility before the field does. Its defining move is doubting the inputs, systematically.
Example¶
An NGO is powering a cash-transfer experiment: does an unconditional monthly stipend raise small-business revenue in participating villages? The headline power calculation says 85% — but it assumes 5% attrition, full take-up, and a baseline revenue spread pulled from a neighboring study. The evaluation team doesn't trust any of those to the decimal, so they build a grid. Rows vary attrition (5/15/25% — migration is common), columns vary take-up (100/85/70% — some assigned households never collect), and a third layer varies the baseline variance (as reported, and 50% higher). The grid is unnerving: under the optimistic corner, power is the promised 85%; but at 25% attrition with 70% take-up and higher variance, it craters to about 40%. The design's advertised power lives entirely in its best-case corner. That finding redirects the budget — toward tracking movers and boosting take-up — rather than toward the extra villages a naive reading would have bought.
How it works¶
- Fix the effect, list the doubts. Hold the meaningful effect constant; enumerate the nuisance parameters whose true values are uncertain — variance, attrition, compliance, clustering, baseline rate.
- Build the grid. Choose a small set of plausible values for each (typically a base case plus a pessimistic and an optimistic bracket) and form the combinations.
- Recompute power in every cell. Feed each combination to a base power engine and record the resulting power, producing a table or heatmap of outcomes.
- Find the cliffs. Locate where power falls below target and identify which parameter movement drives the collapse — that is the assumption the design is hostage to.
Tuning parameters¶
- Parameters gridded — which nuisance inputs get their own axis; more axes map fragility more fully but multiply cells combinatorially.
- Range width per axis — how pessimistic the worst bracket is; too narrow hides the cliff, too wide dilutes the grid with implausible corners.
- Grid resolution — how many values per axis; finer grids locate the cliff edge precisely at more compute.
- Aggregation rule — whether the design is judged by its worst cell, its median, or a probability-weighted average across cells; the worst-cell rule is the most conservative.
- Base-engine choice — whether each cell is filled by a formula or a simulation; simulation lets the grid reach complex designs at higher cost.
When it helps, and when it misleads¶
Its strength is that it kills optimistic-assumption lock-in: it refuses to let a design rest on a single flattering input and shows exactly which uncertainty the whole power story hangs on, so the redesign can target that parameter directly. It misleads when the grid's own ranges are drawn optimistically — a matrix that never brackets a truly bad variance manufactures false comfort, robustness theater over a landscape that was never allowed to look grim. In clustered designs the parameter that most often does the damage is the intraclass correlation, whose plausible range is wide and whose effect on power is severe, so a grid that pins it at one hopeful value misses the real risk.[n1] The guarding discipline is to bracket each nuisance parameter honestly — including values that would embarrass the design — and to judge the design by its credible worst cell, not its comfortable center.
How it implements the components¶
sensitivity_scenario_grid— it is the grid: a matrix of power outcomes across combinations of rival assumptions, the archetype's guardrail against idealized inputs.noise_and_variance_profile— variance and clustering are among the axes it sweeps, so the design's dependence on the assumed noise level is made explicit.attrition_and_missingness_allowance— dropout and compliance are gridded directly, pricing how much sensitivity erodes as retention falls.
It does not build an operating_characteristic_model — it consumes a base engine (Closed-Form Power Calculation or Simulation-Based Power Analysis) to fill each cell. Its nearest twin is Operating Characteristic Curve: the curve varies the true effect and owns the decision_relevant_effect_threshold, whereas this grid holds the effect fixed and varies the nuisance assumptions — the opposite axis of doubt.
Related¶
- Instantiates: Hypothesis Test Power Calibration — the robustness check that stress-tests a design's assumptions.
- Consumes: Closed-Form Power Calculation or Simulation-Based Power Analysis — fills each grid cell; Pilot Variance Estimation anchors the base case its ranges bracket around.
- Sibling mechanisms: Closed-Form Power Calculation · Simulation-Based Power Analysis · Minimum Detectable Effect Table · Operating Characteristic Curve · Pre-Analysis Power Statement · Pilot Variance Estimation
Editorial Notes¶
Form Classification¶
Form family: Analysis, Modeling & Optimization
Rationale: Power Sensitivity Grid operates as an analytical, modeling, inference, comparison, or optimization procedure that derives insight or a solution because it recomputes power across a grid of alternative variance, attrition, and compliance assumptions to expose designs that only clear the bar under optimistic inputs.
Independent corroboration: The frozen evidence defines Power Sensitivity Grid as 'Recomputes power across a grid of alternative variance, attrition, and compliance assumptions to expose designs that only clear the bar under optimistic inputs', so its operative form is Analysis, Modeling & Optimization.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Statistics & Experimental Design
Origin pattern: Single lineage
Present-day reach: Multi-domain
Rationale: Recomputing statistical power across alternative design assumptions is a standard sensitivity-analysis practice in experimental design.
Review resolution: Both blind reviewers agree that statistics experimental design is the primary origin. Reconciliation resolves domain reach disagreement. Formative alternate lineages are retained as none; later breadth of use is recorded separately as domain_reach=multi_domain, while origin_mode=single_lineage describes the relationship among origin lineages.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
[n1] The intraclass correlation measures how alike units within the same cluster are. In cluster-randomized designs it drives the design effect that inflates required sample size, and its plausible range is often wide while its impact on power is steep — making it the parameter a sensitivity grid most needs to bracket honestly rather than pin at a single hopeful value. ↩