Skip to content

Shadow-Map Evaluation

Evaluation process — instantiates Context-Keyed Representation Switching

Runs a candidate map in parallel on live inputs with its outputs suppressed, promoting it to active only once it demonstrably matches or beats the incumbent.

Shadow-Map Evaluation is how a new or updated map earns the right to go live. Before switching a context onto the candidate, you run it in shadow: it receives the same live inputs as the active map, but its outputs are discarded — it acts on nothing. You compare candidate against incumbent on real traffic and promote only if it wins. Its distinguishing property is that it is a live but no-effect trial: it tests an update against the actual current context distribution, at zero blast radius, which an offline comparison of static maps cannot do and a real cutover cannot do safely.

Example

A payments platform keeps context-specific fraud-scoring maps — per region, per merchant category. A new model for the cross-border e-commerce context is ready, and switching it live outright is exactly the move that has burned the team before. Instead, they run Shadow-Map Evaluation: every cross-border transaction is scored by both the incumbent and the candidate, only the incumbent's score actually acts, and the candidate's scores are logged and compared. Over two weeks of real traffic the candidate catches roughly 12% more fraud at the same false-positive rate, on live cases the incumbent had missed — evidence that clears the promotion guard. Had the candidate instead quietly over-blocked good customers, shadow mode would have shown it with zero customer harm, and it would never have been promoted.

How it works

  • Mirror live inputs to the candidate map alongside the active one.
  • Suppress the candidate's outputs — log them, never act on them.
  • Compare candidate against incumbent on real outcomes over enough traffic to be conclusive.
  • Gate promotion: switch the candidate to active only if it meets pre-set criteria; otherwise discard it.

Tuning parameters

  • Shadow duration / sample — how much live traffic before deciding. Longer is more conclusive, slower, and more compute.
  • Promotion criteria — the bar the candidate must clear (parity, or a defined margin, on chosen metrics). Stricter means safer and fewer promotions.
  • Comparison metric — what "better" means for this context — recall, latency, agreement rate. The wrong metric will bless a bad map.
  • Traffic coverage — shadow all traffic or a representative slice. Full coverage catches rare contexts; a slice saves compute.

When it helps, and when it misleads

Its strength is that it de-risks a map update by testing it on real context distributions with zero effect before it can act, catching regressions an offline test misses precisely because it uses live, current traffic;[1] it makes promotion evidence-based rather than calendar-based. Its failure modes come from what a suppressed observer cannot see: shadow only exercises what current traffic happens to contain, so rare or adversarial contexts may never appear in the window; and because the candidate's outputs are suppressed, it cannot reveal effects that only manifest once the map actually acts — feedback loops, or actors gaming the new behaviour. A badly chosen metric can also pass a map that is worse on what matters. The classic misuse is running shadow just long enough to rubber-stamp a decision already made, or reading a noisy near-tie as a win. The discipline is to fix the promotion criteria and the duration before looking at results, and to treat "no evidence of harm" as weaker than "evidence of benefit."

How it implements the components

Shadow-Map Evaluation fills the guarded-promotion subset — how a candidate map is proven before it is switched in:

  • map_update_and_reconsolidation_policy — shadow evaluation is the evidence-gathering gate of the update policy: a candidate is promoted (reconsolidated as the active map) only after it passes the shadow trial.
  • switch_transition_guard — promotion to active is the guarded switch, and the shadow results are the guard's admission criterion, blocking any candidate that has not earned the cutover.

It does not compare two static map snapshots offline (that's Map Difference and Integrity Check), route a real slice of live traffic onto the new map to test it in effect (that's the Canary Context Switch), or preserve and version the maps it compares (that's Versioned Map Registry).

Notes

Shadow evaluation (a no-effect parallel run) and the Canary Context Switch (a small real slice actually served by the new map) are the two live promotion tests, and they trade off oppositely: shadow risks nothing but cannot observe action-effects, while canary observes real effects but puts the slice at risk. They are often used in sequence — shadow first, then canary, then full switch — each admitting the map a little further.

References

[1] Shadow deployment (shadow mode) — the practice of running a new component on production inputs with its outputs discarded, to compare it against the incumbent under real load before it is allowed to serve. Used here exactly as the promotion gate for a candidate map. ↩