Skip to content

Prediction Error Review

Periodic error review — instantiates Predictive Residual Processing

A standing review where people sit with the material misses — building the story of why each gap happened and deciding whether the model, the data, the action, or the boundary should change.

Automation can detect that predictions missed; it cannot authorize what to do about it. Prediction Error Review is the recurring human ritual that adjudicates accumulated prediction errors: for each material gap between expectation and outcome, it reconstructs why it happened — was the model wrong, the data bad, a one-off event, or an assumption that no longer holds — tags the provenance of what it examined, and decides what, if anything, changes. Its product is an explained, sourced verdict and a change decision, not a signal or a score. It is the governance loop that turns scattered residuals into deliberate learning and settles the question no detector can: does this surprise mean fix the model or the world moved?

Example

Every month, an FP&A team runs a budget-variance review, sitting with the largest gaps between forecast and actual P&L. For each material line they reconstruct the cause: was the revenue model structurally wrong, did a data feed arrive late or misclassified, was there a genuine one-off (a lost customer, a weather event), or has a planning assumption — a "boundary" — quietly stopped being true? Each variance gets a written trace and a disposition tagged with its provenance (which forecast version, which data source): revise the model, fix the feed, accept as noise, or change the assumption. Those decisions, with their reasoning attached, feed the next cycle — so the review compounds into better forecasting rather than a monthly autopsy.

How it works

  • Gather the material errors — the gaps that clear a materiality bar, drawn from monitoring, backtesting, and flags.
  • Build the explanatory trace for each — the reconstructed causal story, written before the verdict to blunt hindsight.
  • Tag provenance — model version, data source, whether the input was raw or reconstructed — so each error is anchored to what actually produced it.
  • Classify and decide — model / data / action / boundary problem, then authorize a change or accept.

Tuning parameters

  • Materiality threshold — which gaps make the agenda; too low and the review drowns, too high and slow rot is missed.
  • Cadence — how often it runs, trading freshness against ritual fatigue.
  • Who is in the room — owners of the model, the data, and the action; the review is only as good as its authority to change things.
  • Trace depth — how far each cause is chased before a disposition.
  • Disposition taxonomy — the fixed set of outcomes (model/data/action/boundary) that forces a clear call.

When it helps, and when it misleads

Its strength is converting scattered errors into structured learning and into the model/data/boundary changes automation is not allowed to make on its own.

Its failure modes are cognitive. Hindsight bias makes every past miss look obvious in retrospect, inviting a tidy just-so story; outcome bias judges the original decision by how it happened to turn out.[1] The classic misuse is running the review to assign blame or to rationalize the existing model rather than to test it. The discipline: write the trace before the verdict, tag provenance so claims are checkable, and track whether prior review decisions actually improved later predictions.

How it implements the components

  • explanatory_trace — its core deliverable: the reconstructed, written why-story behind each material error.
  • provenance_marker — it tags each reviewed error with its origin — model version, data source, raw-vs-reconstructed — so verdicts rest on what actually produced the miss.

It does not detect the errors live (that's Model Drift Monitoring), test residual structure statistically (that's Residual Comparison Test), or perform the model update it authorizes (that's Bayesian Model Update).

Notes

The review decides; it does not execute. It authorizes a change and then hands the mechanical work off — the model revision to Bayesian Model Update, a threshold edit to Confidence Threshold Table. Keeping adjudication separate from execution is what lets the same forum govern many downstream mechanisms.

References

[1] Hindsight bias (the miss looks predictable after the fact) and outcome bias (judging a decision by its result rather than its reasoning) are the standard threats to any error post-mortem; writing the causal trace before rendering the verdict is a recognized corrective. ↩