Shadow-Mode Evaluation¶

Deployment-adjacent evaluation — instantiates Model-Based Regulation

Runs a candidate policy silently on live inputs with zero authority to act, logging what it would have done so its divergences from reality can gate promotion.

The riskiest moment for a model-based regulator is the first time a new policy is allowed to act on the real system. Shadow-Mode Evaluation removes that gamble by running the candidate live but powerless: it sees the real production input stream and produces its recommendations and predictions in real time, yet it is given zero authority to actuate anything. Every would-be decision is logged next to what the incumbent system actually did and what then happened, so the candidate accumulates a track record against genuine production reality before it is ever trusted with control. Its defining move is severing authority from evaluation — the candidate earns promotion on evidence gathered at zero operational risk, because at no point can a mistaken recommendation reach the world.

Example¶

A carmaker has a new automatic-emergency-braking policy that brakes earlier for cyclists. Rather than ship it, they run it in shadow mode across the fleet: on every drive it computes, in real time, whether and when it would brake — but the brakes stay under the current production system. Each would-brake event is logged alongside what the human driver and the live system actually did, and whether a real hazard was present.

Over ≈3 months the log tells a two-sided story: the candidate would have caught a handful of genuine near-misses the incumbent missed, but it also would have triggered roughly one phantom brake per 900 miles — too many to promote yet. The candidate never touched a brake pedal; the evidence to reject this version and iterate was bought entirely at zero risk. Only when a later revision's shadow log shows the near-miss catches without the phantom braking does it earn the right to actuate.

How it works¶

What distinguishes shadow mode from an offline test is that it evaluates on the live distribution while withholding all authority:

Tap the real input stream and run the candidate on it in parallel with production, at production tempo.
Sever authority — the candidate's outputs are recorded, never actuated; the severance is enforced, not merely promised.
Pair every output with reality — log the candidate's would-be decision next to the actual decision and the realized outcome, so you measure counterfactual performance ("would this have been better?"), not mere agreement.
Accumulate until decisive, then promote, reject, or iterate — the log is the gate.

Tuning parameters¶

Authority setting — strictly recommend-only vs. allowed to act on a tiny reversible slice (a canary); pure shadow is zero-risk but blind to how its own actions would reshape the system, a canary sees those feedback effects at some risk.
Traffic coverage — run the shadow on all events or a sample; full coverage catches rare cases, sampling saves compute.
Promotion threshold — how much better, over how long, before the candidate earns control; strict thresholds delay good policies, loose ones promote lucky ones.
Comparison basis — score against the incumbent's decisions, against human decisions, or against realized outcomes; outcome-based is truest but lags, since outcomes arrive late.
Operator exposure — whether humans see the shadow's recommendation; showing it invites automation bias, hiding it forgoes an early human sanity check.

When it helps, and when it misleads¶

Its strength is gathering real-world, real-distribution evidence about a candidate at zero operational risk — the one thing offline backtests and simulations cannot fully supply, because they cannot reproduce the true live input distribution. It makes promotion an evidence-based, reversible decision instead of a leap of faith.

Its structural blind spot is feedback: shadow mode cannot see the effects of its own actions, because it takes none. A policy that would have changed the system's trajectory — braking reshapes traffic, a price change reshapes demand — looks fine in shadow precisely because the world it observed was shaped by the incumbent, not by it. The signature misuse is surfacing shadow recommendations to operators "just for information," which quietly steers their decisions via automation bias^[1] and contaminates the very comparison being run; a close second is running shadow only long enough to bless a launch already decided. The discipline is to keep authority genuinely severed, and for policies with strong feedback effects to graduate to a small controlled canary rather than trusting shadow alone.

How it implements the components¶

Shadow-Mode Evaluation fills the pre-promotion assurance side of the archetype — observing a candidate against reality without building, estimating, or attacking it:

authority_scope — it defines and enforces the candidate's authority as zero: recommend and predict only, no actuation.
decision_trace — for each live event it records what the candidate would have decided, beside the actual decision and the outcome.
audit_trail — the accumulated paired record is the durable evidence base on which promotion is granted or refused, and later reviewed.

It does not model the system (State-Space Model), infer its live state (Bayesian State Estimation), or adversarially break the candidate (Model-Failure Red Team); it withholds authority and lets reality score the candidate.

Instantiates: Model-Based Regulation — supplies the zero-risk promotion gate a new policy must clear before it may act.
Consumes: a candidate policy — e.g. from Model Predictive Control — whose real-world behaviour it evaluates.
Sibling mechanisms: Residual-Monitoring Dashboard · Champion–Challenger Evaluation · Model-Failure Red Team · Historical Replay · Digital Twin Trial

Notes¶

Shadow-Mode Evaluation and Champion–Challenger Evaluation both pit a candidate against an incumbent, but they split on authority. Champion–challenger lets the challenger actually act on a slice of live traffic (a controlled split), so it observes feedback effects — at some real risk. Shadow mode withholds all authority, so it risks nothing but stays blind to feedback. The natural order is shadow first to catch the cheap failures, then a champion–challenger split or canary once the shadow log looks clean.

References¶

[1] Automation bias is the well-documented tendency of people to over-rely on automated recommendations and under-weight contradicting evidence. It is why even a strictly "recommend-only" output, if shown to operators, can still steer their decisions — and why the safest shadow evaluation keeps its recommendations out of the operators' view. ↩