Skip to content

Historical Replay

Retrospective testing method — instantiates Model-Based Regulation

Reruns a candidate policy over real recorded history to see what it would have decided, then measures those counterfactual decisions against what actually happened.

Historical Replay takes a candidate policy and runs it back over the system's own recorded past — the real logs, cases, or traces — to reconstruct what it would have decided at each moment, and then sets those counterfactual decisions beside what actually happened. Its evidence is neither synthesised nor curated: it is the genuine history the system already lived through, replayed decision by decision. That is what separates it from a simulated trial, which invents conditions, and from scenario testing, which hand-picks a few. Replay confronts the policy with the full, messy, representative record of reality as it actually unfolded, and asks where the policy would have done better or worse.

Example

A hospital runs an early-warning rule that flags patients sliding toward sepsis. A revised rule is proposed that watches lactate trend alongside the existing vitals. Before it drives a single real alert, it is replayed over three years of de-identified ICU records: for every past admission the system reconstructs exactly when the new rule would have fired (its counterfactual decision trace), then lines that up against the recorded outcome — when sepsis was actually confirmed, and when the original rule fired (the ground-truth feedback already in the logs).

The replay shows the revised rule would have flagged roughly a fifth of the eventually-septic patients about six hours earlier, at the price of some extra early alerts on patients who never became septic. That residual — earlier true catches weighed against added false alarms, computed over real patients — is the evidence the clinical-governance board actually deliberates on, and it costs nothing but a query over data the hospital already holds. What replay cannot tell them is how clinicians would have reacted to those earlier alerts — a limit the board notes explicitly before approving a small live trial.

How it works

Replay earns its keep on one commitment — real history, run through honestly:

  • Reconstruct the counterfactual decisions. For each past case the policy is re-executed on the inputs available at that time, producing the trace of what it would have done, with strict discipline against letting later information leak in.
  • Score against recorded outcomes. Those counterfactual decisions are compared with what actually happened and with the old policy's actions, yielding a residual measured over the true population rather than a chosen sample.
  • Respect the acted-upon past. The record only holds outcomes for the actions actually taken; replay reads the gap it can see and flags the gap it cannot — how the world would have answered the new policy's different actions.

Tuning parameters

  • History window — how far back and across which regimes the replay reaches. Longer windows are more representative but blend in eras when the system behaved differently.
  • Point-in-time discipline — how strictly inputs are restricted to what was knowable at each moment. Loosening it leaks hindsight and flatters the policy; tightening it is realistic but harder to engineer.
  • Comparison baseline — replay against the old policy's actions, against realized outcomes, or both. Each answers a different question — relative improvement versus absolute quality.
  • Residual metric — what "better" means: earlier detection, fewer false alarms, lower cost. The choice quietly decides which policy looks best.
  • Case weighting — whether rare, high-stakes cases are up-weighted so they aren't drowned out by routine ones.

When it helps, and when it misleads

Its strength is that it is cheap, representative, and grounded in reality — it grades a policy on the actual population it will face, using data you already have, before any live exposure.

Its deep limit is that logged history only records the outcomes of the actions that were actually taken, so replay cannot truly observe how the world would have responded to the new policy's different choices — the missing-counterfactual problem at the heart of off-policy evaluation.[1] In practice this surfaces as distribution shift (a policy that would have steered the system somewhere history never went is scored on the wrong data) and as the standing temptation to let future information leak backwards into a past decision, which manufactures a winner. The discipline is point-in-time honesty, treating replay as an offline screen rather than proof, and confirming survivors with a guarded live trial where the counterfactual actually plays out.

How it implements the components

Historical Replay fills the retrospective-evidence slice — the components that turn recorded history into a judgment about a policy:

  • prediction_outcome_residual — its core product: the gap between what the policy would have decided and what actually happened, measured over the real record.
  • decision_trace — it reconstructs, case by case, the trace of decisions the policy would have produced, kept point-in-time correct.
  • feedback_signal — it mines the real outcomes already sitting in the logs as the ground-truth feedback those counterfactual decisions are scored against.

It does not generate novel or synthetic conditions (Digital Twin Trial), curate boundary cases (Scenario Testing), or decide which model then reigns (Champion–Challenger Evaluation).

Notes

Historical Replay is easily confused with Forecast Backtesting. Backtesting asks whether the model's predictions matched later reality; replay asks what the policy's decisions would have produced. A model can forecast well yet drive poor decisions — or forecast badly yet decide well because the decision is insensitive to the error — so the two evaluate different links in the chain and are complementary rather than interchangeable.

References

[1] Evaluating a policy from data generated by a different policy — off-policy evaluation — is limited by the fact that the log holds outcomes only for the actions that were taken; the outcome of a road not taken is unobserved. This is a version of the fundamental problem of causal inference, and it is why a replay that looks decisive still gates rather than replaces live testing. ↩