Posterior-Predictive Residual Check¶
Model-checking method — instantiates Residual-Driven Model Refinement
Simulates replicated datasets from the fitted model and asks whether the observed residuals look like data the model itself would produce.
The Posterior-Predictive Residual Check makes a model grade its own homework by simulation. Its defining move is where the reference comes from: instead of comparing residuals to a textbook distribution, it draws many replicated datasets from the fitted model's posterior predictive distribution, computes the same residual or discrepancy statistic on each replicate, and asks where the observed value falls within that model-generated spread. If the model routinely produces data unlike what was actually seen, the observed discrepancy lands in the tail of its own simulated distribution — the model cannot reproduce a feature of reality. Because the yardstick is manufactured by the model itself, the check works in the model's own units and can interrogate features — overdispersion, zero-inflation, multimodality — that no closed-form residual test even names.
Example¶
Researchers fit a Bayesian Poisson model of disease case counts across several hundred districts. The ordinary residuals look unremarkable, so the model seems to pass. They then run a posterior-predictive check on a targeted discrepancy: the number of districts reporting zero cases. Simulating roughly a thousand replicated datasets from the fitted model, they find each replicate produces only about twenty to forty zero-count districts — but the real data has around a hundred and twenty. The observed count sits far outside the model's own envelope. The Poisson simply cannot manufacture that many zeros.
That gap is a directed diagnosis: the missing structure is excess zeros — overdispersion or a distinct "no-outbreak" process the single-rate Poisson ignores. The team extends the model to a zero-inflated (or negative-binomial) form and re-runs the identical check; now the observed zero-count settles comfortably inside the replicated spread. The model's own simulations exposed a defect that no residual mean or variance would have revealed.
How it works¶
- Pick a discrepancy that targets a feature you care about — a tail count, a maximum, the variance, the number of zeros, a serial correlation. The check is only as sharp as the statistic chosen, and the statistic is fixed before looking.
- Simulate from the posterior predictive, integrating over parameter uncertainty rather than a single point estimate; this is what separates the method from a plug-in bootstrap and keeps the envelope honest.
- Locate the observed statistic within the distribution of replicated statistics, summarized as a posterior-predictive p-value or, better, as an overlaid graphical envelope.
- Read a deep-tail value as a directed target — the model cannot reproduce that specific feature, which is exactly the structure the next revision should add.
Tuning parameters¶
- Discrepancy statistic — which feature of the data is tested; the single highest-leverage choice, and the one most important to commit to before seeing the result.
- Parameter-uncertainty integration — full posterior draws versus a plug-in point estimate; only the former produces an envelope that honestly reflects what the model claims.
- Number of replicates — how many datasets are simulated; more sharpens the estimate of where the observed value sits in the tail, at a compute cost.
- Graphical versus p-value summary — an overlaid distribution of the statistic versus a single number; the picture resists the false crispness a lone p-value invites.
When it helps, and when it misleads¶
Its strength is reach: it can interrogate any feature of the data, not just the mean and variance, and it grounds the word "surprising" in the model's own generative claims rather than an assumed error law. That makes it the natural check for complex, hierarchical, or non-Gaussian models where the textbook residual diagnostics simply do not apply.
Its central caution is subtle. The observed data were used both to fit the model and to check it, so posterior-predictive p-values are conservative — a "pass" is weak evidence, because the model has already been bent toward the very data it is being tested against.[n1] And, as with every diagnostic here, a discrepancy chosen after seeing the residuals can be steered to pass or to fail. The classic misuse is reporting a comfortable p-value from a statistic that could never have failed, in order to bless a model already chosen. The discipline is to pre-commit the discrepancy to the feature that genuinely matters, to read a failing check as strong and a passing one as weak, and to confirm any resulting revision on genuinely held-out data.
How it implements the components¶
reference_explanation_specification— the fitted generative model is the reference; the whole premise is that a good explanation should be able to regenerate the data it purports to explain.uncertainty_and_noise_envelope— it constructs that envelope by simulation, so the question "how much deviation is just noise?" is answered by the model itself rather than by an assumed distribution.
It does not choose the residual arithmetic for a simple plot or test distributional shape against a fixed law — that is Quantile-Quantile Residual Check — nor localize the misfit to a predictor or a subgroup, which are Residual-versus-Fitted Plot and Subgroup Residual Heatmap.
Related¶
- Instantiates: Residual-Driven Model Refinement — it supplies a model-generated adequacy verdict for features the closed-form residual tests cannot reach.
- Sibling mechanisms: Quantile-Quantile Residual Check · Residual-versus-Fitted Plot · Residual Comparison Test · Autocorrelation and Whiteness Test · Heteroscedasticity and Scale Test · Subgroup Residual Heatmap · Residual Root-Cause Review
Editorial Notes¶
Form Classification¶
Form family: Analysis, Modeling & Optimization
Rationale: Posterior-Predictive Residual Check operates as an analytical, modeling, inference, comparison, or optimization procedure that derives insight or a solution because it simulates replicated datasets from the fitted model and asks whether the observed residuals look like data the model itself would produce.
Independent corroboration: The frozen evidence defines Posterior-Predictive Residual Check as 'Simulates replicated datasets from the fitted model and asks whether the observed residuals look like data the model itself would produce', so its operative form is Analysis, Modeling & Optimization.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Statistics & Experimental Design
Origin pattern: Single lineage
Present-day reach: Multi-domain
Rationale: Simulating replicated residuals from a fitted Bayesian model is a standard posterior-predictive diagnostic in statistics.
Review outcome: Independent reviewer agreement; high confidence.
Notes¶
The method needs a generative model — one that can simulate replicate data. A bare point-prediction model with no error component cannot produce replicates, so this check simply does not apply to every model the refinement loop meets; for those, the plot- and law-based diagnostics carry the load instead.
[n1] The posterior-predictive p-value locates an observed discrepancy within the distribution of that discrepancy over data replicated from the fitted model. Because the same data inform both the fit and the check, it is known to be conservative — biased toward the middle — which is why a passing check should be read as weak evidence and a failing one as strong. ↩