Pilot Variance Estimation¶
Empirical estimation — instantiates Hypothesis Test Power Calibration
Runs a small pilot to measure the variance, baseline rate, and dropout that every power calculation depends on, replacing guessed nuisance parameters with data.
The single idea that makes this mechanism itself and not the calculators it feeds: it measures the noise inputs from real data rather than computing anything about power. Every power calculation, formula or simulation, needs numbers it cannot itself supply — how variable the outcome is, what the baseline rate looks like, how many participants will drop out. Guess these and the whole power story rests on sand. Pilot Variance Estimation runs a small, deliberately under-powered study — not to detect the effect, but to observe the nuisance parameters — and returns an empirical variance, baseline, and attrition rate that the downstream power tools consume. It answers none of the questions the family exists to answer (it never estimates power or sample size); it supplies the raw material those questions depend on, and it doubles as a feasibility rehearsal of recruitment, measurement, and retention. Its defining commitment is empirical: where its siblings assume or sweep the noise, this one counts it.
Example¶
A consumer health startup plans a randomized trial of whether a new coaching feature raises daily activity in its fitness app. The power calculation stalls on one unknown: how variable are users' daily step counts, really? Rather than borrow a spread from a published cohort that may not match their users, they run a two-week pilot with a few dozen volunteers — no treatment, just measurement. The pilot returns what they needed: a daily step standard deviation of roughly 2,800 steps, a baseline median near 6,500, and — the number they hadn't planned for — a 20% dropout by day fourteen as novelty faded. It also surfaces a feasibility snag: the wearable sync failed for a handful of users, a measurement gap they can now fix before the real study. Those three empirical numbers, especially the sobering attrition, flow straight into the sample-size calculation, which now rests on observed behavior rather than a hopeful borrow.
How it works¶
- Design for measurement, not detection. The pilot is sized to estimate nuisance parameters, not to test the hypothesis; it is expected to be under-powered for the effect and that is fine.
- Observe the noise directly. Compute the outcome's variance, baseline rate, and correlation structure from the pilot sample, in the same units and instrument the real study will use.
- Watch the dropout. Record how many participants are lost and when, and whether missingness looks random or outcome-related — an attrition estimate no formula can invent.
- Rehearse feasibility. Note what broke — recruitment pace, measurement failures, burden complaints — so the confirmatory design fixes it, and hand the estimated parameters downstream.
Tuning parameters¶
- Pilot size — more pilot participants sharpen the variance estimate but spend budget the main study needs; too few and the estimate is itself wildly uncertain.
- Pilot duration — long enough to observe realistic attrition and any drift, short enough not to become a study in its own right.
- Parameters targeted — variance only, or also baseline rate, correlation, and dropout; each added target needs adequate pilot data to pin down.
- Estimate stance — reporting the point estimate versus a conservative upper bound on variance; the conservative choice guards against underpowering the real study.
- Instrument match — how closely the pilot's measurement matches the planned one; any mismatch makes the borrowed numbers answer a different design.
When it helps, and when it misleads¶
Its strength is grounding: it replaces a borrowed or guessed variance and an optimistic zero-attrition assumption with numbers observed in the actual population and instrument, and it catches operational failures while they are still cheap to fix. Its central failure mode is that a small pilot's variance estimate is itself noisy and tends to read low, so taking the point estimate at face value quietly underpowers the confirmatory study — the reason methodologists recommend carrying a conservative upper bound on the pilot variance rather than its raw estimate into the sample-size calculation.[n1] The classic misuse is a different sin entirely: mining the pilot for a treatment effect and reporting it, when the pilot was never powered to detect one. The guarding discipline is to use the pilot only for nuisance parameters and feasibility, to inflate the variance estimate for its own uncertainty, and to resist reading an effect into a sample that cannot support one.
How it implements the components¶
noise_and_variance_profile— its primary output: an empirical variance, baseline rate, and correlation measured from real observations rather than assumed.attrition_and_missingness_allowance— it observes actual dropout and its timing and pattern, supplying an attrition estimate the calculators would otherwise have to guess.feasibility_and_ethics_constraint— the pilot is a live rehearsal that surfaces recruitment, measurement, and burden problems before the full study commits resources.
It builds no operating_characteristic_model and computes no power — it feeds the numbers that Closed-Form Power Calculation and Simulation-Based Power Analysis turn into power. Its nearest twin is Power Sensitivity Grid, which also handles noise and attrition; but the grid owns the sensitivity_scenario_grid, sweeping a range of assumed nuisance values, whereas this mechanism measures a single empirical value from real data.
Related¶
- Instantiates: Hypothesis Test Power Calibration — the empirical front end that supplies the noise inputs the calibration needs.
- Sibling mechanisms: Closed-Form Power Calculation · Simulation-Based Power Analysis · Minimum Detectable Effect Table · Operating Characteristic Curve · Power Sensitivity Grid · Pre-Analysis Power Statement
Editorial Notes¶
Form Classification¶
Form family: Experiment, Test & Rehearsal
Rationale: Pilot Variance Estimation operates as an active test, trial, simulation, drill, or rehearsal that generates evidence through a deliberate attempt or perturbation because it runs a small pilot to measure the variance, baseline rate, and dropout that every power calculation depends on, replacing guessed nuisance parameters with data.
Independent corroboration: The frozen evidence defines Pilot Variance Estimation as 'Runs a small pilot to measure the variance, baseline rate, and dropout that every power calculation depends on, replacing guessed nuisance parameters with data', so its operative form is Experiment, Test & Rehearsal.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Statistics & Experimental Design
Origin pattern: Single lineage
Present-day reach: Specialized
Rationale: Pilot Variance Estimation is rooted in experimental design and statistics: Statistical power analysis uses pilot estimates of variance, baseline rate, and dropout to size later studies.
Review outcome: Independent reviewer agreement; high confidence.
Notes¶
Pilot Variance Estimation is deliberately upstream of every calculator in this family: it produces inputs, never a power verdict. Keeping it separate is what lets a team improve its noise estimate — a larger or longer pilot — without re-running the design logic that consumes it, and what keeps the honest warning intact that a pilot sizes the noise, not the effect.
[n1] A widely cited methodological caution: variance estimated from a small pilot is imprecise and tends to be biased downward, so using its raw point estimate to size a study systematically underpowers it. The standard remedy is to carry a one-sided upper confidence limit on the pilot variance — a deliberately conservative value — into the sample-size calculation. ↩