Predictive Replication Check¶
Simulation — instantiates Distributional-Assumption Governance
Simulates replicate datasets from the fitted model and asks whether they reproduce the observed shape and dependence the decision relies on.
If a model is a good description of the world, then data it generates should look like the data it was fit to — at least on the features that matter. Predictive Replication Check puts that logic to work: it draws many replicate datasets from the fitted distributional commitment, then compares those synthetic datasets to the real one on decision-relevant features — the height of the peaks, the frequency of zeros, the clustering of high values, the co-movement of variables. If the fitted model cannot even generate its own data's important features, it is misspecified in a way that no single-number fit statistic reveals. This is an in-sample self-consistency test: it does not withhold data or score out-of-sample performance. Its signature strength is catching the failure where every marginal looks fine while the joint structure — dependence, clustering, tail co-movement — is wrong, because marginal fit statistics are blind to exactly that.
Example¶
A public-health team fits a Poisson model to weekly dengue case counts across a city's districts, to size vector-control crews. In-sample, the mean counts match and the fit looks acceptable. The replication check simulates hundreds of replicate count-series from the fitted Poisson and compares them to the observed data on three features the staffing decision depends on: the size of outbreak peaks, the number of zero-case weeks, and the spatial clustering of high-count districts.
The replicates fail on all three. Simulated peaks are far too tame, there are too few zero weeks, and the synthetic districts show no spatial clustering, whereas the real data cluster sharply during outbreaks. The diagnosis is immediate: Poisson cannot produce the overdispersion and spatial dependence the real epidemic has, so a crew plan built on it will under-staff exactly the clustered outbreaks it is meant to cover. The team moves to a negative-binomial model with a spatial term — caught before the staffing commitment was made, and caught specifically because the check looked at joint features rather than the marginal average.
How it works¶
- Simulate from the fitted commitment. Many replicate datasets are drawn under the model's own parameters — the model is asked to reproduce reality.
- Choose discrepancy features that matter. The comparison targets decision-relevant summaries — tail counts, zero mass, cluster sizes, dependence — not generic goodness of fit.
- Locate the observed among the replicates. If the real feature sits far in the tail of the replicate distribution, the model cannot reproduce it.
- Aim at joint structure. The check is at its most valuable on dependence and clustering, the features marginal diagnostics cannot see.
Tuning parameters¶
- Discrepancy features — which summaries are compared. Choosing only features the model is built to match guarantees a pass and proves nothing.
- Number of replicates — how many synthetic datasets are drawn. More stabilizes the reference distribution at more compute.
- Graphical vs. test — an eyeball comparison of replicate versus observed, or a formal tail-probability. Graphs surface surprises; a single p-value invites absolutism.
- In-sample vs. mild look-ahead — pure posterior-predictive replication versus replication on lightly held-back data, trading self-consistency for a touch of out-of-sample bite.
When it helps, and when it misleads¶
Its strength is catching marginal-fit / joint-failure: a model whose one-variable summaries all pass but which cannot generate the dependence or the tails the decision leans on. The idea is the posterior predictive check — simulate data under the fitted model and see whether the real data look like a plausible draw — and its power is precisely that it tests the whole generative story, not one moment of it.[1]
Its failure mode is in-sample optimism: because the same data fit the model and judge it, a model can reproduce its own training features and still fail on genuinely new data, so a pass here is necessary but not sufficient. And a check is only as honest as its features — quietly choosing summaries the model is guaranteed to match turns replication into self-congratulation. The discipline that keeps it honest is to pick discrepancy features from the decision, not from the model, to include the joint and tail features that are most likely to break, and to treat a pass as "not obviously self-inconsistent," with out-of-sample coverage tested separately.
How it implements the components¶
empirical_shape_and_diagnostic_profile— the replicate-versus-observed comparison is a diagnostic profile aimed at whether the fitted model reproduces the shape features (peaks, zeros, spread) the decision uses.joint_dependence_and_conditional_structure— its defining contribution is checking whether simulated data reproduce clustering, serial, spatial, and tail dependence, which correct marginals never guarantee.
This check works in-sample and never scores the model on data withheld from fitting against coverage bars — that out-of-sample validation (simulation_predictive_and_holdout_validation, fit_for_use_acceptance_and_exception_thresholds) is the Holdout Calibration and Coverage Backtest.
Related¶
- Instantiates: Distributional-Assumption Governance — replication supplies the self-consistency evidence that guards against joint and tail misspecification.
- Consumes: Support, Shape, and Tail Diagnostic Suite — the observed shape features it compares the replicates against.
- Sibling mechanisms: Holdout Calibration and Coverage Backtest · Resampling Robustness Audit · Support, Shape, and Tail Diagnostic Suite · Distributional Sensitivity Grid
Draft mechanism page for the Encyclopedia of Abstractions.
Editorial Notes¶
Form Classification¶
Form family: Analysis, Modeling & Optimization
Rationale: Predictive Replication Check operates as an analytical, modeling, inference, comparison, or optimization procedure that derives insight or a solution because it simulates replicate datasets from the fitted model and asks whether they reproduce the observed shape and dependence the decision relies on.
Independent corroboration: The frozen evidence defines Predictive Replication Check as 'Simulates replicate datasets from the fitted model and asks whether they reproduce the observed shape and dependence the decision relies on', so its operative form is Analysis, Modeling & Optimization.
Nearest alternative: Experiment, Test & Rehearsal — Predictive Replication Check includes features of an active test, trial, simulation, drill, or rehearsal that generates evidence through a deliberate attempt or perturbation, but its defining operation is an analytical, modeling, inference, comparison, or optimization procedure that derives insight or a solution.
Review outcome: Independent reviewer agreement; medium confidence.
Origin Attribution¶
Primary origin: Statistics & Experimental Design
Origin pattern: Single lineage
Present-day reach: Multi-domain
Rationale: Generating replicate datasets from a fitted model and comparing their shape with observations is a statistical predictive-check method.
Review outcome: Independent reviewer agreement; high confidence.
References¶
[1] A posterior predictive check (Rubin, 1984; Gelman, Meng & Stern, 1996) simulates replicate datasets from a fitted model and compares chosen features of those replicates to the observed data; a feature the model cannot reproduce is evidence of misspecification. It tests the model's whole generative story rather than a single summary statistic. withdrawn registry ↩