Skip to content

Posterior or Simulation Predictive Check

Simulation-based model check — instantiates Stochastic Process Modeling and Validation

Simulates replicate datasets from the fitted model and checks whether real-data summaries the model was not tuned on fall inside or outside the simulated spread, exposing misfit the likelihood hides.

A model can score a high likelihood and still generate runs that look nothing like the real world. Posterior or Simulation Predictive Check catches that by using the model's own output as the yardstick: draw many replicate datasets from the fitted (or posterior) model, compute a discrepancy statistic — a feature the model was not directly fit to — and see whether the value observed in the real data sits comfortably inside the cloud of simulated values or out in its tail. An observed feature far in the tail means the model, however well it fit the average, fails to reproduce something real. Its distinguishing idea is that the reference distribution is generated, not assumed — which is what separates it from a calibration transform or a residual test.

Example

An analyst fits a simple scoring model to a season of football matches. It reproduces the average goals per match almost perfectly. But is it a good generative model of matches? Simulate ≈10,000 replicate seasons from the fitted model and, for each, compute features the fit never targeted: the share of 0–0 draws, the longest run of scoreless matches, the most goals in any single game.

Overlaying the real season on those simulated distributions is diagnostic. Average goals lands dead-centre — no news. But the real number of 0–0 draws falls far in the upper tail of the simulated spread: real football has more scoreless matches than the model can produce. That single misfit reveals a missing ingredient — low-scoring matches cluster in a way independent scoring cannot generate — and points to the fix, without ever appearing in the likelihood.

How it works

  • Pick discrepancy statistics that matter. Choose summaries tied to the model's use and not targeted by the fit — extremes, streak lengths, counts, autocorrelation, threshold crossings.
  • Simulate replicates. Generate many datasets from the fitted model; in the posterior form, draw parameters from the posterior first so parameter uncertainty flows into the replicates.
  • Locate the observed value. Place the real statistic within the replicate distribution — a predictive p-value or, better, a graphical overlay.
  • Read the tail as falsification. An observed statistic in the extreme tail of what the model can generate is evidence the model is structurally wrong on that feature.

Tuning parameters

  • Discrepancy statistic(s) — the entire power of the check lives here; a statistic the model happens to reproduce proves nothing, so choose features that would break if the model is wrong.
  • Number of replicate simulations — more replicates resolve the tail of the reference distribution but cost compute.
  • Posterior-predictive vs plug-in — whether replicates fold in parameter uncertainty (posterior) or fix parameters at a point estimate (plug-in); the former is honest, the latter cheaper.
  • Graphical vs numerical — an eyeball overlay surfaces the shape of misfit; a p-value compresses it to one number, losing detail.
  • Attention threshold — how far into the tail counts as a problem worth acting on.

When it helps, and when it misleads

Its strength is turning "does this model generate realistic runs?" into concrete, targeted evidence, catching structural misfit — missing clustering, wrong extremes, absent streaks — that aggregate likelihood or a good mean fit completely conceal.

Its subtlest failure is double use of the data: checking the model on the very data used to fit it makes the reference distribution too accommodating, so the predictive p-value is conservative and a bad model can slip through by being graded on its own homework.[^ppp] The cruder failure is cherry-picking a statistic the model is bound to pass and declaring victory. The discipline is to choose discrepancy statistics before looking, prefer held-out or out-of-sample replicates where feasible, and run several complementary statistics rather than one flattering one.

How it implements the components

Posterior or Simulation Predictive Check fills the generative-validation side of the archetype's machinery:

  • generated_path_validation_plan — it is a generated-run check: simulate whole datasets and compare their features (counts, streaks, extremes) against the observed run.
  • stochastic_diagnostic_and_falsification_suite — an observed statistic in the tail of the model's own replicates is a falsification test the model can fail.

It does not verify point-by-point calibration of the predictive distribution — that is Probability Integral Transform Check — rank rival models — that is Proper Scoring Rule Comparison — or hunt residual autocorrelation, which is Residual Independence and Whiteness Test; it consumes a fitted model rather than building one.

Notes

Because the same data both fits and checks the model, treat a passed posterior predictive check as weaker evidence than a failed one. A failure is a genuine red flag — the model cannot generate a feature the world shows — whereas a pass can be flattered by the shared data. Where the data budget allows, holding out a slice for the check, or using cross-validated replicates, restores most of the lost bite.