Parametric Bootstrap Coverage Audit¶
Method — instantiates Coverage Probability Calibration
Fits a model to the real data, treats the fitted parameters as ground truth, and generates pseudo-datasets from that model to check whether the interval procedure covers under model-implied conditions.
Parametric Bootstrap Coverage Audit borrows the truth from a model fit to the data at hand. You estimate a parametric model on your real dataset, freeze its fitted parameters, and then pretend those parameters are true: you generate many pseudo-datasets by drawing from the fitted model, rebuild the interval on each, and count how often it contains the fitted parameter value. Its distinguishing idea is that the ground truth is neither invented by the analyst nor awaited from the world — it is inherited from a credible model of the data. That makes it the right tool when you trust the model's shape but doubt whether the interval formula behaves well at your finite sample size. The audit is only as sound as the fitted model; if the model is wrong, the pseudo-datasets are wrong, and the coverage number is confidently misleading.
Example¶
A reliability engineer fits a Weibull lifetime model to 60 observed failure times of a pump seal and reports a 90% confidence interval for the tenth-percentile life (the "B10" the warranty is priced on). The maximum-likelihood interval leans on large-sample theory, but 60 failures is not large, and the estimand sits out in the lower tail where approximations are weakest. To audit it, she freezes the fitted Weibull shape and scale as the pretend-truth, simulates 5,000 fresh datasets of 60 failures from that fitted model, and on each rebuilds the same B10 interval by the same procedure.
Counting containment of the fitted B10, the nominal-90% interval covers only about 82% of the pseudo-datasets — the procedure is optimistic in the tail at this sample size. Because the truth was inherited from the credible Weibull fit rather than assumed by a formula, the audit tells her specifically that her pipeline, on her sample size, at her estimand, undercovers. She flags the B10 interval as needing a wider or corrected construction before it feeds the warranty math.
How it works¶
- Fit and freeze. Estimate the parametric model on the real data; the fitted parameter vector becomes the pretend-truth and pins down exactly what the interval must contain.
- Regenerate under the model. Draw pseudo-datasets of the same size from the fitted model — the regimes are model-implied, not hand-chosen and not the raw data replayed.
- Rerun the whole interval procedure. Apply the identical estimation, transformation, and variance steps to each pseudo-dataset, so the audit grades the real pipeline rather than a textbook stand-in.
- Count containment of the fitted parameter. The hit fraction against the frozen truth estimates coverage under the model's own world.
Tuning parameters¶
- Model choice — which parametric family you fit; a richer family reduces misspecification risk but adds variance and can itself distort the tail behavior you are auditing.
- Number of pseudo-datasets — more draws sharpen the coverage estimate at compute cost, the same Monte-Carlo-error trade every resampling audit faces.
- Nested vs. single-level — whether each pseudo-dataset re-estimates the model (double bootstrap) or reuses the original fit; nesting corrects the audit's own optimism but is markedly more expensive.
- Where the estimand sits — auditing a central parameter versus a tail quantile; tail estimands need far more draws and are where model misspecification bites hardest.
When it helps, and when it misleads¶
Its strength is that it stress-tests the actual interval procedure at the actual sample size using a truth you have some real evidence for — the fitted model — rather than a fiction you typed in. When the model form is credible and only finite-sample behavior is in doubt, it is more grounded than a hand-built simulation and more structured than raw resampling.
Its failure mode is inherited misspecification: the audit certifies coverage under the fitted model[1], so if the real world departs from that model — extra dispersion, contamination, the wrong tail — a procedure can pass the audit and still undercover in practice. This is the subtle trap that separates it from assumption-light resampling: the parametric bootstrap can be fooled by exactly the model error it cannot see. The guarding discipline is to check the fit before trusting the audit (residual diagnostics, an over-dispersed alternative) and to treat a pass as conditional on the model, not on reality.
How it implements the components¶
- interval_procedure_inventory — by rerunning the entire estimation-and-interval pipeline on each model-generated pseudo-dataset, it audits the actual procedure — variance estimator, transformation, and all — not an idealized formula.
- target_quantity_and_estimand_definition — freezing the fitted parameter as the pretend-truth forces an exact, operational definition of the estimand the interval must contain (here, the tenth-percentile life).
It does not invent ground truth from hand-specified regimes — that designed simulation is Monte Carlo Coverage Simulation, its nearest twin, which owns coverage_performance_test — and it does not resample the raw data without a model, which is resampling_benchmark under Nonparametric Resampling Interval Check. It measures the gap but does not act on it: the calibration_adjustment_rule and coverage_width_tradeoff_policy belong to Calibration-Set Interval Adjustment.
Related¶
- Instantiates: Coverage Probability Calibration — it audits whether the model-based interval keeps its coverage promise at finite sample size.
- Sibling mechanisms: Monte Carlo Coverage Simulation · Nonparametric Resampling Interval Check · Finite-Sample or Exact Interval Check · Pre-Registered Simulation Grid · Subgroup Coverage Calibration Table · Calibration-Set Interval Adjustment · Historical or Holdout Coverage Backtest
Editorial Notes¶
Form Classification¶
Form family: Experiment, Test & Rehearsal
Rationale: Parametric Bootstrap Coverage Audit operates as an active test, trial, simulation, drill, or rehearsal that generates evidence through a deliberate attempt or perturbation because it fits a model to the real data, treats the fitted parameters as ground truth, and generates pseudo-datasets from that model to check whether the interval procedure covers under model-implied conditions.
Independent corroboration: The frozen evidence defines Parametric Bootstrap Coverage Audit as 'Fits a model to the real data, treats the fitted parameters as ground truth, and generates pseudo-datasets from that model to check whether the interval procedure covers under model-implied conditions', so its operative form is Experiment, Test & Rehearsal.
Nearest alternative: Assessment, Review & Assurance — Parametric Bootstrap Coverage Audit includes features of a bounded evaluation of existing evidence or work that produces a finding or disposition, but its defining operation is an active test, trial, simulation, drill, or rehearsal that generates evidence through a deliberate attempt or perturbation.
Review outcome: Independent reviewer agreement; medium confidence.
Origin Attribution¶
Primary origin: Statistics & Experimental Design
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Specialized
Rationale: Parametric Bootstrap Coverage Audit is most directly rooted in experimental design and statistics' methods for comparison, uncertainty, sampling, sensitivity, and inferential validation. The lineage fits its defining practice: Fits a model to the real data, treats the fitted parameters as ground truth, and generates pseudo-datasets from that model to check whether the interval procedure covers under model-implied conditions.
Related originating lineages:
- Data Science & Analytics — Parametric Bootstrap Coverage Audit also draws materially on data science and analytics' computational practice of modeling, monitoring, validation, and pattern extraction, which shaped this mechanism rather than merely adopting it as an application.
Review resolution: Both independent reviews agree on primary origin statistics_experimental_design; reconciliation resolves alternate_origin_disagreement, origin_mode_disagreement. Formative alternate lineages retained: data_science. The broader reach of later applications is kept separate as domain_reach=specialized; origin_mode=cross_disciplinary_synthesis records how the formative lineages relate. Confidence is conservatively reconciled to high, and encyclopedia_synthesis=false preserves the reviewers' boundary judgment.
Review outcome: Reconciled after independent review; high confidence.
References¶
[1] Efron, B., and Tibshirani, R. J. An Introduction to the Bootstrap. Chapman & Hall (1993). States that a parametric-bootstrap coverage assessment is conditional on the fitted model. registry ↩