Forecast Backtesting Review¶
Retrospective audit — instantiates Reference-Class Planning Calibration
After a project closes, compares what was forecast to what actually happened, records the signed error, and fires a recalibration so the next plan inherits the correction.
An organization can run a beautiful calibration gate on every project and still stay miscalibrated forever, because nobody ever checks whether the calibrated forecasts came true. Forecast Backtesting Review is the closing-of-the-loop step: after a project finishes, it pulls the original forecast — the estimate, the reference class it used, the adjustment applied, the reserve set — and lays it against the actual outcome, computing the signed error (how much, in which direction) rather than filing a vague "we ran late." Its defining move is that the error is not just noted but routed: a persistent record of forecast-versus-actual accumulates, and when that record shows a systematic bias, the review trips a trigger that changes how the next forecast is made. It is the only mechanism in the family that looks backward at completed forecasts; it does not size a plan, hold the case library, or gate a commitment.
Example¶
A clinical research lab forecasts how long patient recruitment will take for each of its trials — a number that drives staffing, funding milestones, and publication timelines. Historically the coordinators estimate from the inside view: eligibility criteria, referring clinics, expected consent rate. A Forecast Backtesting Review is instituted at trial close-out. For a just-completed cardiology study, it retrieves the original forecast (nine months to full enrolment, no reference class used) and the actual (fourteen months), records the error as +56% with the dominant driver tagged as slower-than-assumed referral flow. Doing this across the last eight closed trials, the review finds the errors are not scattered noise — they are almost all positive, clustering around a persistent +40–60% recruitment overrun. That pattern is the signal that fires the trigger: the lab's default recruitment estimate is systematically optimistic, and the recommendation is to stop forecasting recruitment from the inside view alone and route every new trial's estimate through the class of its own completed studies. The next protocol's timeline is built on that correction, not on fresh optimism.
How it works¶
- Reconstruct the original forecast. Retrieve the estimate as it stood at commitment, with its class, adjustment, and reserve — not a memory of it, the recorded version.
- Measure the signed error. Compute actual minus forecast, as a percentage and with direction, and attribute the dominant cause. Direction is the whole point: a debiasing loop needs to know whether estimates run high or low.
- Accumulate, don't discard. Add the error to a standing record so that single-project noise resolves, over many projects, into a bias signal or a spread.
- Trip the trigger on a pattern. When the accumulated record crosses a threshold — a persistent one-sided bias, or a variance wider than the reserves assumed — flag that the forecasting method itself needs adjustment, and hand that correction to the next planning cycle.
The review deals only in closed projects and completed forecasts; its currency is hindsight made systematic.
Tuning parameters¶
- Review trigger — every project at close-out, a periodic sweep, or only above a stakes threshold. Reviewing everything builds the richest error record but taxes teams already moving on.
- Error metric — raw signed percentage, absolute error, or a calibration score across many forecasts. A one-sided metric exposes bias; a symmetric one rewards precision but can hide a consistent lean.[n1]
- Recalibration threshold — how large or persistent a bias must be before the trigger fires. Too sensitive and it chases noise; too lax and a real bias runs uncorrected for years.
- Attribution depth — whether the review records only the number or also the dominant driver. Drivers make the correction actionable but invite blame narratives that distort the record.
When it helps, and when it misleads¶
Its strength is that it is the archetype's memory and its immune system at once: without it, every plan can repeat the same exceptionalist story, because there is no accumulating evidence that the story keeps being wrong. A signed-error record turns "we always seem to run late" into a measured bias with a size and a direction, which is exactly what a recalibration needs to act on. Tracking calibration across many forecasts — the discipline behind scoring rules like the Brier score — is what separates a genuinely improving forecaster from a lucky one.[n1]
It misleads when the review is run as a blame ritual rather than a calibration one: if a positive error is read as someone's failure rather than the method's bias, teams learn to forecast defensively (pad quietly, redefine "done" after the fact) and the error record is corrupted at the source. It also misleads when scope drift is not separated out — a project that "overran" only because its target changed will poison the bias signal. The guarding discipline is to backtest the forecast, not the people, and to reconcile every error against the scope trace before entering it, so recalibration responds to genuine estimation bias and not to a moving goalpost.
How it implements the components¶
forecast_outcome_memory— its central artifact: a durable, per-forecast record of estimate, class, adjustment, actual, and signed error, so past optimism cannot be forgotten.rolling_recalibration_trigger— when the accumulated errors show a systematic bias, it fires the signal that changes how the next forecast is made, closing the calibration loop.
It does not hold the library of comparable completed cases or their outcome distribution (base_rate_distribution, reference_class_boundary) — that persistent store is the Historical Project Outcome Database, which this review feeds. This review looks back at one project's own forecast versus its own result; the database aggregates many projects into a class.
Related¶
- Instantiates: Reference-Class Planning Calibration — provides the outcome memory and the recalibration trigger that keep the whole gate improving.
- Consumes: Historical Project Outcome Database — the review's completed forecasts and errors become new entries in it.
- Sibling mechanisms: Historical Project Outcome Database · Reference-Class Forecasting Workbook · Launch or Commitment Readiness Gate · Contingency Reserve Formula · Schedule and Cost Risk Register · Independent Estimate Round · Three-Point Estimate with Base Rates · Premortem as Auxiliary Probe
Editorial Notes¶
Form Classification¶
Form family: Assessment, Review & Assurance
Rationale: Forecast Backtesting Review operates as a bounded evaluation of existing evidence or work that produces a finding or disposition because it after a project closes, compares what was forecast to what actually happened, records the signed error, and fires a recalibration so the next plan inherits the correction.
Independent corroboration: The frozen evidence defines Forecast Backtesting Review as 'After a project closes, compares what was forecast to what actually happened, records the signed error, and fires a recalibration so the next plan inherits the correction', so its operative form is Assessment, Review & Assurance.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Behavioral Economics
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Multi-domain
Rationale: Behavioral economics is primary because forecast review exposes hindsight, overconfidence, and incentive-distorted judgment by comparing prior predictions with outcomes. Statistical backtesting, organizational learning, and foresight practice materially shape the review; the generalized artifact is encyclopedia synthesis.
Related originating lineages:
- Futurism & Strategic Foresight — Reference-class forecasting and forecast learning materially frame the retrospective.
- Organizational & Management Science — Post-project comparison of planned and actual outcomes is rooted in management review and project controls.
- Statistics & Experimental Design — Signed error and recalibration supply quantitative validation.
Review resolution: Behavioral economics is primary because forecast review exposes hindsight, overconfidence, and incentive-distorted judgment by comparing prior predictions with outcomes. Statistical backtesting, organizational learning, and foresight practice materially shape the review; the generalized artifact is encyclopedia synthesis.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Researched adjudication after independent review; high confidence.
Sources consulted:
Notes¶
[n1] The Brier score measures the calibration of probabilistic forecasts by averaging squared differences between predicted probabilities and outcomes; the general practice of scoring forecasts against realized results, over many cases, is what lets a forecaster detect and correct a systematic bias rather than mistaking a run of luck for skill. ↩a ↩b