Pre-Registered Simulation Grid¶
Artifact — instantiates Coverage Probability Calibration
A committed-in-advance table of the sample sizes, effect sizes, distributions, dependence structures, missingness, and selection paths a coverage study will test — fixed before any method is run.
A Pre-Registered Simulation Grid is not a test — it is the specification the tests must obey, written down and frozen before anyone sees which method wins. It enumerates the regimes over which coverage will be evaluated: the axes (sample size, effect size, distribution shape, dependence, missingness rate, selection path) and the specific cells of their cross-product that will be run. Its defining purpose is procedural honesty: by committing to the regimes in advance, it removes the analyst's freedom to keep hunting for the friendly corner where a favored interval happens to cover. It is an input artifact — other mechanisms consume it and do the simulating — and its value is entirely in having been fixed before the results, not after.
Example¶
A biostatistics group is preparing a methods paper comparing three confidence-interval procedures for a rare-event rate. To keep the comparison from becoming an exercise in flattering their own preferred method, they publish a simulation grid before running anything: sample sizes {30, 100, 500}, true event rates {0.5%, 2%, 10%}, three distributional shapes, two dependence structures, and two missingness mechanisms — every cell of the cross-product listed explicitly, with the number of replications per cell and the coverage-reporting format fixed. They register it, following the ADEMP structure for simulation studies, so the aims, data-generating mechanisms, estimands, methods, and performance measures are all named up front.
When the results come in, one procedure undercovers badly at n = 30 with the 0.5% rate — a cell the team might have quietly dropped had it not been pre-committed. Because the grid was frozen, that unflattering cell is reported alongside the rest, and the paper's coverage claims are credible precisely because the regimes were not chosen to produce them.
How it works¶
- Name the axes. List every regime dimension that could affect coverage: sample size, effect size, distribution, dependence, missingness, selection and stopping paths.
- Enumerate the cells. Fix the specific cross-product of values to be run, plus replications per cell, so the scope of the claim is explicit and complete.
- Freeze before results. Register or time-stamp the grid before any method is evaluated, removing the degrees of freedom that enable cherry-picking.
- Hand it off. The grid is consumed by the running mechanisms — it defines what to simulate; it does not do the simulating.
Tuning parameters¶
- Grid resolution — how finely each axis is sampled; a fine grid catches narrow failure pockets but multiplies run cost and can bury the signal in cells.
- Coverage of the cross-product — full factorial vs. a screened or fractional design; full is exhaustive but expensive, fractional is cheaper but can miss interaction effects between regimes.
- Adversarial inclusion — how many deliberately hostile cells (boundary parameters, heavy tails, near-total missingness) are committed; more adversarial cells make a pass more meaningful but a clean sweep less likely.
- Amendment policy — whether and how the grid may be revised after registration; any post-hoc change must be logged as such or the pre-registration guarantee is lost.
When it helps, and when it misleads¶
Its strength is that it neutralizes regime cherry-picking — the failure mode where a method looks calibrated only because it was tested where it happens to work. A frozen, public grid makes the scope of a coverage claim auditable[1] and forces the unflattering cells into the report.
Its failure mode is a false sense of completeness: a grid is only as honest as the regimes its authors thought to include, and a pre-registered grid that omits the real deployment regime is confidently, permanently blind to it — pre-commitment does not make an incomplete specification complete. The classic misuse is registering a grid stacked with easy cells, or quietly amending it after peeking. The guarding discipline is to derive the axes from where the interval will actually be used, include deliberately adversarial cells, and log any amendment as an amendment.
How it implements the components¶
- relevant_regime_set — the grid is the relevant-regime set made concrete and enumerable: the committed list of conditions under which coverage must hold.
- coverage_reporting_note — by fixing the reporting format and forcing every registered cell (including failures) into the write-up, it standardizes the honest coverage note that accompanies the results.
It runs nothing itself: the known-truth simulation over these regimes is coverage_performance_test, performed by Monte Carlo Coverage Simulation, which consumes this grid. And it is not the after-the-fact stratified report — that is Subgroup Coverage Calibration Table, its sibling-artifact twin, which owns subgroup_coverage_slice and empirical_coverage_gap_diagnostic; this grid pre-specifies regimes rather than tabulating realized gaps.
Related¶
- Instantiates: Coverage Probability Calibration — it is the pre-commitment artifact that keeps the calibration study from being gamed.
- Sibling mechanisms: Subgroup Coverage Calibration Table · Monte Carlo Coverage Simulation · Parametric Bootstrap Coverage Audit · Nonparametric Resampling Interval Check · Finite-Sample or Exact Interval Check · Calibration-Set Interval Adjustment · Historical or Holdout Coverage Backtest
Editorial Notes¶
Form Classification¶
Form family: Representation, Specification & Plan
Rationale: Pre-Registered Simulation Grid operates as a static representation, map, specification, schema, or prospective plan that externalizes information because it a committed-in-advance table of the sample sizes, effect sizes, distributions, dependence structures, missingness, and selection paths a coverage study will test — fixed before any method is run.
Independent corroboration: The frozen evidence defines Pre-Registered Simulation Grid as 'A committed-in-advance table of the sample sizes, effect sizes, distributions, dependence structures, missingness, and selection paths a coverage study will test — fixed before any method is run', so its operative form is Representation, Specification & Plan.
Nearest alternative: Rule, Policy & Commitment — Pre-Registered Simulation Grid includes features of a standing rule, threshold, contractual commitment, or policy constraint governing future conduct, but its defining operation is a static representation, map, specification, schema, or prospective plan that externalizes information.
Review outcome: Independent reviewer agreement; medium confidence.
Origin Attribution¶
Primary origin: Statistics & Experimental Design
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Specialized
Rationale: Pre-specifying distributions, sample sizes, missingness, and selection paths for a coverage study is statistical simulation design and preregistration.
Related originating lineages:
- Operations Research — Scenario-grid and computational experiment design materially shape systematic exploration of complex conditions.
Review resolution: Both blind reviewers agree that statistics experimental design is the primary origin. Reconciliation resolves alternate origin disagreement, origin mode disagreement, encyclopedia synthesis disagreement. Formative alternate lineages are retained as operations_research; later breadth of use is recorded separately as domain_reach=specialized, while origin_mode=cross_disciplinary_synthesis describes the relationship among origin lineages.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Reconciled after independent review; high confidence.
References¶
[1] Morris, T. P., White, I. R., & Crowther, M. J. "Using simulation studies to evaluate statistical methods". Statistics in Medicine 38(11), 2074–2102 (2019). Provides a structured simulation-study framework that makes the scope of design and performance evaluation transparent and auditable. registry ↩