Off-Policy Evaluation¶
Analysis method — instantiates Policy Evaluation Before Deployment
Estimates how a candidate policy would perform directly from logged data generated by a different policy, correcting statistically for the fact that the logs were never collected under the candidate.
Off-Policy Evaluation answers "how well would this new rule do?" without running it and without building a simulator — it estimates the candidate's value straight from logs that a different policy produced. The defining problem it solves is that those logs are biased: every action in them was chosen by the incumbent (the behavior policy), so the data over-represents what the incumbent liked and under-represents what the candidate would do. The mechanism's whole job is to correct that bias arithmetically. It reweights each logged outcome by how much more or less likely the candidate was to take that same action, so a record collected under one policy yields an honest estimate of a policy that never ran. That statistical correction is exactly what separates it from a naïve rerun of history: it does not just replay the candidate over the past, it reweights the past to stand in for a world the candidate would have created.
Example¶
A music-streaming service has a candidate policy for ordering the "Up Next" autoplay queue, meant to lift track-completion rate. It cannot A/B test every idea on live listeners, but it holds months of logged sessions from the current ranking policy: for each session, which track was shown, whether the listener finished or skipped it, and — crucially — the probability the logging policy assigned to that choice (the propensity).
Off-Policy Evaluation reweights those logs. A session where the candidate would have made the same pick the incumbent happened to make counts heavily; a session where the two diverge is down-weighted, because it says little about the candidate. Summed with an importance weight of candidate-probability ÷ logged-probability per action, and stabilized with a small reward model to tame the variance, the estimate reads: about +3% completion rate overall, with a confidence interval — and sliced by cohort, negative for brand-new listeners whose taste the logs barely cover. That single subgroup finding, produced from data the service already owns at zero listener exposure, is what sends the candidate back for revision rather than to launch.
How it works¶
- Reweight, don't replay. Each logged outcome is multiplied by an importance weight comparing how likely the candidate and the logging policy were to take that action; the weighted average estimates the candidate's expected outcome.
- Require propensities or model them. The correction needs the logging policy's action probabilities. Where they were recorded, the estimate is honest; where they must be reconstructed, that reconstruction becomes a new source of bias.
- Stabilize with a direct model. Doubly-robust variants combine the reweighting with a fitted reward model, so the estimate stays trustworthy if either the weights or the model are right — a hedge against the reweighting's high variance.
- Slice the estimate. Because reweighting is just a weighted sum, it can be run within any subgroup to expose where an aggregate gain hides a localized loss.
Tuning parameters¶
- Estimator choice — inverse-propensity, direct-method, or doubly-robust. More correction lowers bias but raises variance; the direct method is stable but trusts its model.
- Weight clipping — capping extreme importance weights. Clipping tames variance but reintroduces bias, trading a wild estimate for a quietly shaded one.
- Propensity source — logged vs. reconstructed action probabilities. Logged is faithful; reconstructed is often the only option and the weakest link.
- Overlap requirement — how much the candidate is allowed to diverge from the logging policy. Demanding strong overlap keeps estimates tight but rules on fewer candidates.
- Subgroup granularity — how finely the estimate is sliced. Finer slices catch localized harm but thin the data per slice until the intervals blow out.
When it helps, and when it misleads¶
Its strength is cheap, zero-exposure evidence: it grades a candidate on the real population using data you already have, returns a confidence interval rather than a point, and slices to catch the subgroup harm an average conceals.
Its deep limit is overlap. Where the candidate would do things the logging policy almost never did, the logs hold little relevant evidence, importance weights explode, and the estimate's variance balloons — the "curse of horizon" that makes long-sequence off-policy estimates notoriously unstable.[n1] A candidate that steers the system somewhere the incumbent never went is exactly where the estimate is least trustworthy and most confidently wrong. The classic misuse is reporting the point estimate while hiding the effective sample size, so a high-variance guess masquerades as a verdict. The discipline is to publish the weight distribution and effective sample size alongside the number, refuse estimates where overlap is too thin, and treat the result as an offline screen that a guarded live trial must still confirm.
How it implements the components¶
Off-Policy Evaluation fills the estimate-from-existing-evidence slice — turning logs into a value judgment without generating anything new:
outcome_metric— its core product is a counterfactual estimate of the outcome metric (here, completion rate) for a policy that never ran.comparison_baseline— the logging/behavior policy is the built-in baseline; the estimate is naturally expressed as the candidate's incremental value over the rule that generated the data.subgroup_or_context_slice— the same reweighting is run within cohorts, exposing where aggregate lift masks a per-group loss.
It does not build a forward simulator or construct a scenario and edge-state set — trajectory_evaluation_model, scenario_or_trace_set, and edge_state_catalog are filled by Policy Simulation; nor does it package the decision — deployment_context_boundary, deployment_gate, evaluation_evidence_log, and rollback_or_monitoring_handoff belong to Simulation-Based Validation Report.
Related¶
- Instantiates: Policy Evaluation Before Deployment — supplies a value estimate for the candidate before any live exposure.
- Sibling mechanisms: Policy Simulation · Simulation-Based Validation Report · Historical Replay · Scenario Testing · Digital Twin Trial
Editorial Notes¶
Form Classification¶
Form family: Analysis, Modeling & Optimization
Rationale: Off-Policy Evaluation operates as an analytical, modeling, inference, comparison, or optimization procedure that derives insight or a solution because it estimates how a candidate policy would perform directly from logged data generated by a different policy, correcting statistically for the fact that the logs were never collected under the candidate.
Independent corroboration: The frozen evidence defines Off-Policy Evaluation as 'Estimates how a candidate policy would perform directly from logged data generated by a different policy, correcting statistically for the fact that the logs were never collected under the candidate', so its operative form is Analysis, Modeling & Optimization.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Computer Science & Software Engineering
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Multi-domain
Rationale: Reinforcement learning developed off-policy evaluation for estimating a target policy from trajectories generated by a different behavior policy.
Related originating lineages:
- Data Science & Analytics — Logged-decision analytics operationalized counterfactual policy comparison before deployment.
- Statistics & Experimental Design — Causal inference and importance sampling supplied weighting, overlap assumptions, variance analysis, and uncertainty.
Review resolution: Both independent reviews agree on primary origin computer_science; reconciliation resolves domain_reach_disagreement. Formative alternate lineages retained: data_science, statistics_experimental_design. The broader reach of later applications is kept separate as domain_reach=multi_domain; origin_mode=cross_disciplinary_synthesis describes the historical relationship among lineages. Confidence is conservatively reconciled to high, and encyclopedia_synthesis=false preserves the reviewers' boundary judgment.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
Off-Policy Evaluation is easily confused with Historical Replay, and the difference is the whole point of the method. Replay reruns the candidate over recorded history and reads off what it would have decided, taking the logged distribution at face value; Off-Policy Evaluation reweights that history to correct for the fact the logs came from a different policy. Replay tells you what the candidate would have done on the past that happened; off-policy estimation tries to tell you what it would achieve on the future it would itself create — which is why it carries importance weights and replay does not.
[n1] The curse of horizon is the tendency of importance-sampling estimators to accumulate variance multiplicatively over long decision sequences, so that off-policy estimates for sequential policies become unstable as the horizon grows. It is the standard reason a clean off-policy number still gates rather than replaces a guarded live trial. ↩