Skip to content

Forecast Backtesting

Retrospective validation audit — instantiates Predictive Residual Processing

Replays a predictor against withheld history — across time, segments, and regimes — to earn or deny the right to suppress its residuals.

Before you let a system stop sending the full signal and transmit only what its model failed to predict, you have to know the model is actually good enough — and where. Forecast Backtesting is the retrospective, out-of-sample check that replays a predictor over history it never trained on and asks: across which regimes, segments, and horizons were its predictions close enough that the leftover residual can safely be treated as the whole message? Unlike a live drift watch, it happens once, up front, on frozen history; unlike a single-point anomaly test, it maps the envelope of trust. Its whole job is to convert "the model seems fine" into a bounded claim about where suppression is licensed and where the raw signal must still flow.

Example

A grid operator wants to cut telemetry by having day-ahead demand forecasts sent downstream as residuals only — dispatch reconstructs load from its own copy of the forecast plus the small correction. Before committing, they backtest. They replay the demand model walk-forward over ≈3 years of withheld data, scoring predicted against realized load, sliced by season, weekday/weekend, and by heatwave regime. In normal conditions the residuals are tiny — suppression would save enormously. But during extreme-heat regimes the residual balloons: the model badly under-predicts air-conditioning load. The backtest's verdict isn't a yes/no; it's a scope map — suppress inside the normal-weather envelope, keep the full signal during heat events — which is exactly the information the rest of the loop needs.

How it works

The distinguishing discipline is strict separation of what the model knew from what it was predicting:

  • Walk-forward, out-of-sample. Train on the past, predict the next slice, roll forward — never scoring a period the model could have seen. Look-ahead leakage is the cardinal sin.
  • Slice by segment and regime. Aggregate accuracy hides the regimes where the model fails; backtesting deliberately partitions history to find them.
  • Score realized against predicted and report the envelope — the conditions and horizons where residuals stay inside tolerance — not a single headline error number.

Tuning parameters

  • Window scheme — expanding vs. rolling training window; rolling adapts to recent regimes, expanding uses all history.
  • Segmentation granularity — finer slices expose regime-specific failure but thin the data per slice and invite noise.
  • Regime coverage — whether the rare, dangerous regimes even appear in the available history; absent regimes can't be validated, only flagged as untested.
  • Out-of-sample strictness — embargo gaps and holdout discipline; how hard you work to prevent leakage.
  • Refresh cadence — one-shot pre-launch, or periodic re-backtest as history accumulates.

When it helps, and when it misleads

Its strength is that it separates "good enough to suppress" from "not," and localizes it — the difference between a blanket compression that hides rare events and a scoped one that doesn't. It is the evidence that licenses residual suppression at all.

Its failure modes cluster around fooling yourself: backtest overfitting and data-snooping (tuning knobs until the historical curve looks good, then mistaking that fit for predictive skill), look-ahead leakage, survivorship, and regimes simply missing from history.[1] The classic misuse is running the backtest backward — to bless a model already chosen — until it flatters. The discipline that guards against it: pre-register the protocol, keep one untouched holdout, and treat a passed backtest as necessary but never sufficient — the world keeps moving after the test ends.

How it implements the components

  • prediction_target_definition — you cannot backtest until the target and its realization measure are pinned exactly; the backtest forces that definition to be concrete and scorable.
  • model_scope_and_horizon — the core output is the regime/segment/horizon envelope within which the model's predictions hold well enough to suppress.

It does not size the tolerated residual budget behind a decision (that's Confidence Threshold Table), watch for live degradation after launch (that's Model Drift Monitoring), or maintain and update the model itself (that's Bayesian Model Update).

Notes

A passed backtest licenses suppression only inside the tested envelope and only until the world moves. It is the pre-launch gate; the continuous live watch is a different mechanism — hand that to Model Drift Monitoring, and hand the "why did this slice fail?" diagnosis to Residual Comparison Test.

References

[1] Data-snooping (or data-dredging) and look-ahead bias are the standard names for over-fitting a backtest to historical quirks and for accidentally using information unavailable at prediction time. Out-of-sample, walk-forward evaluation with a held-out final slice is the recognized corrective, which is why it defines the method above. ↩