Historical or Holdout Coverage Backtest¶
Audit — instantiates Survival-Conditioned Persistence Forecasting
Checks whether persistence intervals issued before the outcome was known actually contained the realized lifetimes at their stated rate, catching forecasts that are confident but wrong.
A persistence forecast usually ships with a band — "this will last within X–Y, with 80% probability." Historical or Holdout Coverage Backtest is the check that the band is not a comfortable fiction. It takes a batch of past forecasts, each of which claimed a probability p that the truth would fall inside a stated interval, waits until the outcomes are known (or holds out cases whose outcomes it hides), and counts how often the truth actually landed inside. If 80% intervals contain the outcome ≈80% of the time, they are calibrated; if far less, the forecaster is over-confident. Its defining focus — the thing that separates it from a general accuracy backtest — is that it grades the uncertainty band, not the point estimate: a razor-sharp prediction is worthless if its stated confidence is a lie.
Example¶
A consumer lender's risk team issues, for each loan cohort, an interval: "80% of this cohort's defaults will occur within months X–Y." To find out whether those intervals are honest, they run a coverage backtest on cohorts originated ≈4 years ago whose outcomes are now fully observed. For each, they reconstruct the interval they would have quoted at origination — using only data available then, no peeking ahead — and ask whether the realized default-timing fell inside it. Across ≈40 past cohorts, the nominally-80% intervals contained the outcome only ≈55% of the time. The model was badly over-confident. The fix is not a cleverer point forecast; it is widening the bands until historical coverage matches the label they carry — and logging that the old bands ran hot, so nobody re-trusts them by default.
How it works¶
- Freeze forecasts at issue time — use only information available before the outcome, so the test can't be flattered by look-ahead.
- Wait for resolution, or hold out — score against outcomes that were genuinely unknown when the interval was set.
- Score coverage — the empirical hit rate against the nominal level, aggregated over many cases, not judged one at a time.
- Separate coverage from sharpness — report both; a band can cover perfectly by being uselessly wide.
- Handle still-alive cases explicitly — a right-censored subject tells you only that the truth exceeds a bound, which coverage accounting must respect.
Tuning parameters¶
- Nominal level(s) tested — 50/80/95%; testing several exposes whether miscalibration is uniform or worst in the tails.
- Holdout scheme — strict out-of-time split vs. random holdout; out-of-time is the honest one for a forecasting task.
- Aggregation grain — overall coverage vs. coverage sliced by subgroup or horizon; fine slices catch pockets of over-confidence a global number hides.
- Censoring treatment — how still-alive cases enter the count; mishandling them silently biases coverage.
- Pass/fail tolerance — how far empirical coverage may drift from nominal before the bands are declared broken.
When it helps, and when it misleads¶
Its strength is that it catches the failure a point-accuracy score cannot see — bands that are confident and wrong — using only pre-outcome information, and it keeps calibration and sharpness as separate verdicts so a wide-and-useless forecast can't hide behind good coverage.[n1]
It needs enough resolved cases to say anything, and right-censoring erodes that supply. Its deeper trap is that past coverage guarantees future coverage only if the process is stationary — good historical calibration says nothing once the regime shifts. And it is easily run backwards: tuning intervals against the very cases used to test them until coverage looks good is not calibration, it is leakage. The discipline is a strict out-of-time holdout, reporting sharpness alongside coverage, and re-running as regimes move rather than trusting a one-time pass.
How it implements the components¶
This backtest owns the archetype's calibration-and-memory side — it validates and records, it does not forecast:
uncertainty_and_confidence_band— it grades and recalibrates these bands; checking their coverage is its entire reason to exist.forecast_error_memory— it accumulates the coverage track record, the institutional memory that tells future forecasts how honest past bands turned out to be.
It does not fit the model that produces the intervals — that is Survival or Time-to-Event Analysis — nor locate the regime break that would void them, which belongs to Change-Point Detection and Stationarity Test.
Related¶
- Instantiates: Survival-Conditioned Persistence Forecasting — it is the appraisal's honesty check on its own stated confidence.
- Consumes: Survival or Time-to-Event Analysis and Age-Conditioned Remaining-Life Table supply the interval forecasts it grades.
- Sibling mechanisms: Survival or Time-to-Event Analysis · Forecast Backtesting · Reference-Class Forecasting · Change-Point Detection · Rolling Window Comparison · Periodic Durability Inspection · Stationarity Test · Age-Conditioned Remaining-Life Table · Censoring and Left-Truncation Audit · Hazard-Shape Diagnostic · Lifetime Distribution Comparison · Lindy Decision-Horizon Review · Non-Aging Eligibility Review
Editorial Notes¶
Form Classification¶
Form family: Experiment, Test & Rehearsal
Rationale: Historical or Holdout Coverage Backtest operates as a bounded trial, probe, simulation, or rehearsal that generates evidence from performance because it checks whether persistence intervals issued before the outcome was known actually contained the realized lifetimes at their stated rate, catching forecasts that are confident but wrong
Independent corroboration: The frozen evidence defines Historical or Holdout Coverage Backtest as 'Checks whether persistence intervals issued before the outcome was known actually contained the realized lifetimes at their stated rate, catching forecasts that are confident but wrong', so its operative form is Experiment, Test & Rehearsal.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Statistics & Experimental Design
Origin pattern: Single lineage
Present-day reach: Multi-domain
Rationale: Checking realized coverage against previously issued probability intervals is statistical forecast calibration and verification.
Related originating lineages:
- Economics & Finance — Financial backtesting materially developed institutional coverage tests for probabilistic risk forecasts.
Review resolution: Both reviewers independently assign statistics_experimental_design as the primary originating domain, so that shared primary is retained. Alternate domains are the union of reviewer-identified formative or independently originating lineages; later application settings alone are excluded. The evidence describes one principal historical lineage. It has established independent use across several domains, but that does not make it domain-free. The encyclopedia entry generalizes the established mechanism without creating a new composite lineage.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
Coverage is necessary but not sufficient. A forecast can be perfectly calibrated and still useless if its intervals are enormous — good coverage bought with terrible sharpness. Always read the two numbers together. And keep this distinct from a general Forecast Backtesting pass, which grades point accuracy: a model can nail the median and still lie about its uncertainty, which is exactly the gap this test exists to close.
[n1] In forecast evaluation, calibration means stated probabilities match observed frequencies and sharpness means the intervals are as tight as honesty allows. Coverage backtesting measures the first directly; the standard practice is to maximize sharpness subject to calibration, so neither number is read alone. ↩