Skip to content

Historical or Holdout Coverage Backtest

Audit — instantiates Survival-Conditioned Persistence Forecasting

Checks whether persistence intervals issued before the outcome was known actually contained the realized lifetimes at their stated rate, catching forecasts that are confident but wrong.

A persistence forecast usually ships with a band — "this will last within X–Y, with 80% probability." Historical or Holdout Coverage Backtest is the check that the band is not a comfortable fiction. It takes a batch of past forecasts, each of which claimed a probability p that the truth would fall inside a stated interval, waits until the outcomes are known (or holds out cases whose outcomes it hides), and counts how often the truth actually landed inside. If 80% intervals contain the outcome ≈80% of the time, they are calibrated; if far less, the forecaster is over-confident. Its defining focus — the thing that separates it from a general accuracy backtest — is that it grades the uncertainty band, not the point estimate: a razor-sharp prediction is worthless if its stated confidence is a lie.

Example

A consumer lender's risk team issues, for each loan cohort, an interval: "80% of this cohort's defaults will occur within months X–Y." To find out whether those intervals are honest, they run a coverage backtest on cohorts originated ≈4 years ago whose outcomes are now fully observed. For each, they reconstruct the interval they would have quoted at origination — using only data available then, no peeking ahead — and ask whether the realized default-timing fell inside it. Across ≈40 past cohorts, the nominally-80% intervals contained the outcome only ≈55% of the time. The model was badly over-confident. The fix is not a cleverer point forecast; it is widening the bands until historical coverage matches the label they carry — and logging that the old bands ran hot, so nobody re-trusts them by default.

How it works

  • Freeze forecasts at issue time — use only information available before the outcome, so the test can't be flattered by look-ahead.
  • Wait for resolution, or hold out — score against outcomes that were genuinely unknown when the interval was set.
  • Score coverage — the empirical hit rate against the nominal level, aggregated over many cases, not judged one at a time.
  • Separate coverage from sharpness — report both; a band can cover perfectly by being uselessly wide.
  • Handle still-alive cases explicitly — a right-censored subject tells you only that the truth exceeds a bound, which coverage accounting must respect.

Tuning parameters

  • Nominal level(s) tested — 50/80/95%; testing several exposes whether miscalibration is uniform or worst in the tails.
  • Holdout scheme — strict out-of-time split vs. random holdout; out-of-time is the honest one for a forecasting task.
  • Aggregation grain — overall coverage vs. coverage sliced by subgroup or horizon; fine slices catch pockets of over-confidence a global number hides.
  • Censoring treatment — how still-alive cases enter the count; mishandling them silently biases coverage.
  • Pass/fail tolerance — how far empirical coverage may drift from nominal before the bands are declared broken.

When it helps, and when it misleads

Its strength is that it catches the failure a point-accuracy score cannot see — bands that are confident and wrong — using only pre-outcome information, and it keeps calibration and sharpness as separate verdicts so a wide-and-useless forecast can't hide behind good coverage.[1]

It needs enough resolved cases to say anything, and right-censoring erodes that supply. Its deeper trap is that past coverage guarantees future coverage only if the process is stationary — good historical calibration says nothing once the regime shifts. And it is easily run backwards: tuning intervals against the very cases used to test them until coverage looks good is not calibration, it is leakage. The discipline is a strict out-of-time holdout, reporting sharpness alongside coverage, and re-running as regimes move rather than trusting a one-time pass.

How it implements the components

This backtest owns the archetype's calibration-and-memory side — it validates and records, it does not forecast:

  • uncertainty_and_confidence_band — it grades and recalibrates these bands; checking their coverage is its entire reason to exist.
  • forecast_error_memory — it accumulates the coverage track record, the institutional memory that tells future forecasts how honest past bands turned out to be.

It does not fit the model that produces the intervals — that is Survival or Time-to-Event Analysis — nor locate the regime break that would void them, which belongs to Change-Point Detection and Stationarity Test.

Notes

Coverage is necessary but not sufficient. A forecast can be perfectly calibrated and still useless if its intervals are enormous — good coverage bought with terrible sharpness. Always read the two numbers together. And keep this distinct from a general Forecast Backtesting pass, which grades point accuracy: a model can nail the median and still lie about its uncertainty, which is exactly the gap this test exists to close.

References

[1] In forecast evaluation, calibration means stated probabilities match observed frequencies and sharpness means the intervals are as tight as honesty allows. Coverage backtesting measures the first directly; the standard practice is to maximize sharpness subject to calibration, so neither number is read alone. ↩