Skip to content

Champion–Challenger Evaluation

Evaluation review — instantiates Model-Based Regulation

Runs the incumbent regulating model against candidate challengers on the same objective and promotes a challenger only when it beats the champion by a pre-set margin.

Champion–Challenger Evaluation keeps one model — the champion — in charge of live regulation while one or more challengers run alongside it on the very same inputs and are scored against the very same objective. Nothing about a single model's absolute quality decides anything here; what decides is the contest. A challenger is promoted to champion only when it beats the incumbent by a margin large enough to clear an explicit promotion rule, and otherwise the champion reigns on. That framing — a standing, head-to-head tournament in which a defeated incumbent keeps the throne by default — is what separates it from the one-model tests around it: it is the mechanism that decides which model should be regulating, not whether a given model is any good.

Example

A bank has regulated its lending cut-offs for years with a scorecard everyone calls the champion. A data-science team believes a newer model would approve more good borrowers at the same loss rate, but nobody will hand live lending decisions to an unproven model. So the challenger is deployed beside the champion: every incoming application is scored by both, the champion's decision is the one that actually acts, and the challenger's decision is logged as what it would have done. Both are graded on the same objective — expected loss at a fixed approval volume — over the same applicants.

After ≈9 months the challenger is running roughly 4% lower losses at equal volume. The promotion rule was fixed in advance: a challenger takes the throne only if it beats the champion by at least a set margin sustained over two quarters, on a population large enough that the gap is unlikely to be noise. The challenger clears it, is promoted to champion, and the old scorecard is demoted to challenger status itself — where it keeps running as a live check on its successor.

How it works

What makes this more than an A/B test is the asymmetry and the persistence:

  • The incumbent holds the throne. The champion makes the real decisions throughout; challengers are scored on shadowed or replayed decisions and never trusted with live authority until they win, so the cost of being wrong about a challenger is bounded.
  • One fixed objective, one arena. Every candidate is graded on the same pre-agreed objective over the same population, so the comparison is like-for-like rather than each model reporting its own flattering metric.
  • Promotion is a rule, not a mood. A margin, a duration, and a minimum sample are set before the contest, so a challenger that merely got lucky does not get crowned.
  • The tournament never closes. A promoted challenger becomes the champion others must now beat, and the old champion usually stays on as a challenger. Regulation improves by continual succession rather than one-time replacement.

Tuning parameters

  • Promotion margin — how much better a challenger must be before it takes over. Tight margins churn the champion on noise; wide margins entrench a stale incumbent.
  • Evaluation horizon — how long a challenger must sustain its lead. Longer horizons filter luck and regime blips but slow adoption of genuine gains.
  • Number of challengers — one careful rival or a whole field. More challengers explore faster but multiply the chance that one wins by luck, and cost more to run.
  • Live vs. shadow scoring — whether challengers act on a small traffic slice or are only shadowed. A live slice measures true effect but exposes real decisions to an unvetted model.
  • Champion handicap — whether the incumbent must be beaten outright or by enough to justify switching costs. Encodes how much you value stability over marginal improvement.

When it helps, and when it misleads

Its strength is that regulation can improve continuously without ever betting live authority on an unproven model, and every candidate is forced onto one honest scoreboard instead of self-reporting.

The classic trap is the winner's curse: run enough challengers and one will top the champion by chance alone; promote on that single win and you have installed noise.[1] Two more failure modes shadow it. A fixed objective can be gamed — a challenger that optimizes the measured metric while quietly degrading something unmeasured looks like a winner (Goodhart territory). And the whole apparatus is easily run backwards, with the "contest" staged to anoint a model the team has already decided to ship. The discipline is to fix the promotion rule — margin, duration, sample — before looking at results, keep the retired champion running as a live check on its successor, and watch unmeasured outcomes for the gaming case.

How it implements the components

Champion–Challenger Evaluation fills only the selection-and-objective slice of the archetype — the parts that decide which model rules:

  • regulatory_objective_set — operationalizes the regulator's objective as the single scoring function every candidate is graded on, making the objective explicit and shared rather than implicit and per-model.
  • model_update_rule — is the promotion rule: the pre-set margin, horizon, and sample that govern when the reigning model is replaced, and the demotion path back to the incumbent.

It does not build or score the models it compares: the like-for-like evidence comes from Historical Replay and Shadow-Mode Evaluation, the models themselves from State-Space Model, and the version record of each promotion from Model Registry.

Notes

The method only regulates relative quality — it promotes the best available model, not necessarily a good one. If the reigning champion is itself poor, a challenger can win the throne while still failing the regulator's real-world requirements. Pair it with an absolute bar (safety or scenario acceptance from Scenario Testing) so that "better than the champion" cannot quietly substitute for "good enough."

References

[1] In a field of many candidates evaluated on noisy data, the one that scores best is biased upward — some of its apparent edge is luck that regresses on the next sample. This winner's curse (a form of regression to the mean) is why sound champion–challenger promotion rules demand a margin sustained over a horizon rather than a single-period win. ↩