Off-Policy or Historical Replay Evaluation¶
Offline evaluation method — instantiates Sequential Policy Optimization
Estimates how a proposed policy would have performed by replaying historical logs from the policy that actually ran, reweighted to correct for what the old policy chose to try.
Sometimes you cannot simulate a policy and you dare not test it live — but you have a mountain of logs from whatever policy did run. Off-Policy or Historical Replay Evaluation estimates how a new candidate policy would have performed by replaying those historical traces. Its defining difficulty, and the thing that separates it from a forward simulation, is that the logs were generated by a different policy — the behavior policy — which chose which actions got tried and which states got visited. You never observe what your candidate would have caused; you only observe what the old policy caused. So replay's signature move is reweighting: it corrects for the mismatch between what the candidate would have done and what the logs actually recorded, up-weighting the logged decisions the candidate agrees with and discounting the rest. Done honestly it gives a defensible retrospective estimate; done carelessly it extrapolates confidently into regions the logs never covered.
Example¶
A news app wants to try a new article-ranking policy but cannot A/B test every idea on live users, and there is no faithful simulator of what people click. It has months of logs, though: for each session it recorded the user's context, which articles the old ranker showed, and whether the user clicked. Historical replay estimates the new ranker's click-through by walking those logs. Where the new policy would have shown the same top article the old one happened to show, that logged click (or non-click) is real evidence and counts. But the old ranker showed some articles far more often than others, so those decisions are over-represented; replay reweights each logged outcome by how likely the candidate was to make that choice relative to how likely the behavior policy was — inverse-propensity weighting[n1] — so the estimate reflects the new policy, not the old one's habits.
The estimate is trustworthy exactly where the logs have coverage. For a niche section the old ranker almost never surfaced, the new policy wants to promote articles the logs barely touched — and there replay must say so rather than guess. The outcome is an honest, offline read on the candidate's likely performance, with the low-coverage regions flagged as unknowns rather than papered over.
How it works¶
- Take the logs and the candidate. Use recorded (state, action, reward) traces from the behavior policy, plus the new policy to be scored.
- Reweight by the policy mismatch. Weight each logged outcome by how much more or less likely the candidate was to take that action than the behavior policy — correcting the sampling bias baked into the logs.
- Estimate the candidate's value. Aggregate the reweighted rewards into an expected performance for the new policy.
- Report coverage honestly. Flag the states and actions the logs barely cover, where the estimate is extrapolation rather than evidence.
What distinguishes it is that it is retrospective and model-free: it reuses logged trajectories and corrects for who generated them, never rolling a model forward to make new ones.
Tuning parameters¶
- Weight clipping — capping the correction weights tames the wild variance of rare, heavily-reweighted samples, at the price of some bias; it is the central bias–variance dial.
- Estimator choice — pure importance weighting is unbiased but high-variance; a doubly-robust or model-blended estimator lowers variance if its model is decent, at the cost of a modeling assumption.
- Coverage threshold — how much logged support a state-action must have before its estimate is trusted rather than flagged; stricter yields fewer but more defensible conclusions.
- Behavior-propensity fidelity — how accurately the logging probabilities are known; exact logged propensities give clean corrections, guessed ones inject error into every weight.
When it helps, and when it misleads¶
Its strength is evaluating a policy you can neither simulate nor safely deploy, using data you already have — cheap, fast, and grounded in real outcomes rather than a simulator's assumptions. It is the natural gate before a candidate is allowed anywhere near live traffic.
Its failure mode is the positivity (overlap) assumption[n2]: replay can only speak about state-action regions the behavior policy actually explored, and pushing it to score a candidate that wants to do things the logs never tried yields estimates with either explosive variance or quiet, confident nonsense. The classic misuse is exactly that — trusting a replay number for a policy that departs sharply from the logged one, extrapolating past the data's support. Related traps are unrecorded confounders and mis-stated logging propensities, both of which bias every weight. The guarding discipline is to report coverage explicitly, clip or bound the weights, and refuse to draw conclusions where the logs are thin rather than manufacturing them.
How it implements the components¶
Off-Policy Replay realizes the retrospective, log-based side of policy evaluation:
policy_evaluation_rule— it is a policy-evaluation rule of the offline kind: candidate policies are scored on trajectory-level outcomes drawn from logged experience.reward_cost_function— it aggregates the logged, reweighted rewards into the candidate's estimated value; the recorded reward is the raw material of the estimate.state_observation_model— it must reason explicitly about how states and actions were logged and sampled, since coverage and logging bias determine where the estimate is trustworthy.
It does not roll a transition_model forward to generate fresh trajectories, nor run over a decision_horizon of simulated futures — that prospective, model-based mode is the near-twin Simulation Rollout Evaluation. Replay is retrospective: it reuses the past instead of simulating a future.
Related¶
- Instantiates: Sequential Policy Optimization — the offline, log-based way to estimate a candidate policy's performance.
- Sibling mechanisms: Markov Decision Process Model · Dynamic Programming / Value Iteration · Policy Iteration · Simulation Rollout Evaluation · Reinforcement Learning Policy Learning · Threshold Policy Rule · Adaptive Policy Review Cycle
Editorial Notes¶
Form Classification¶
Form family: Analysis, Modeling & Optimization
Rationale: Off-Policy or Historical Replay Evaluation operates as an analytical, modeling, inference, comparison, or optimization procedure that derives insight or a solution because it estimates how a proposed policy would have performed by replaying historical logs from the policy that actually ran, reweighted to correct for what the old policy chose to try.
Independent corroboration: The frozen evidence defines Off-Policy or Historical Replay Evaluation as 'Estimates how a proposed policy would have performed by replaying historical logs from the policy that actually ran, reweighted to correct for what the old policy chose to try', so its operative form is Analysis, Modeling & Optimization.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Computer Science & Software Engineering
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Multi-domain
Rationale: Sequential decision and reinforcement-learning research developed historical replay and policy evaluation from logs collected under another policy.
Related originating lineages:
- Data Science & Analytics — Production experimentation practice shaped the offline gate used before guarded live rollout.
- Statistics & Experimental Design — Inverse-propensity weighting and causal-identification theory provide correction for action-selection bias and the positivity requirement.
Review resolution: Both independent reviews agree on primary origin computer_science; reconciliation resolves domain_reach_disagreement, encyclopedia_synthesis_disagreement. Formative alternate lineages retained: data_science, statistics_experimental_design. The broader reach of later applications is kept separate as domain_reach=multi_domain; origin_mode=cross_disciplinary_synthesis describes the historical relationship among lineages. Confidence is conservatively reconciled to high, and encyclopedia_synthesis=true preserves the reviewers' boundary judgment.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
[n1] Inverse-propensity weighting reweights each logged outcome by the ratio of the candidate policy's probability of that action to the behavior policy's, correcting for the fact that the logs over-represent whatever the old policy preferred. It is unbiased in principle but high-variance when the ratios grow large. ↩
[n2] The positivity (overlap) assumption requires that every action the candidate might take had some nonzero chance of appearing in the logs. Where it fails — actions the behavior policy never tried — off-policy estimates become extrapolation with no data behind them, and the honest move is to flag the gap rather than report a number. ↩