Skip to content

Champion / Challenger Threshold Test

Controlled-comparison protocol — instantiates Adaptive Threshold Recalibration

Runs a candidate threshold in parallel with the incumbent on the same live traffic and promotes it only if it demonstrably wins.

Champion / Challenger Threshold Test is the protocol that refuses to swap a threshold on a hunch. Its defining move is the head-to-head: the incumbent boundary — the champion — keeps making the real decisions, while one or more challenger thresholds score the identical stream of cases in shadow. Because both see exactly the same inputs, the comparison isolates the effect of the boundary alone, with no confound from a changed population or period. Only if a challenger beats the champion on a pre-committed metric — without tripping a guardrail — does it get promoted. It is the evidence gate between "we think a different cutoff would be better" and actually changing it.

Example

A platform auto-removes comments when a toxicity classifier scores them above 0.90. Moderators suspect the bar is too high — genuinely abusive comments are slipping through. Rather than lower it and hope, they stand up a challenger at 0.80 that scores every incoming comment in shadow: the 0.90 champion still governs what actually gets removed, but the system logs what the 0.80 rule would have done. After two weeks on the same live stream, the challenger would have caught ≈22% more abusive comments while wrongly flagging ≈4% more benign ones — a tradeoff the pre-registered promotion rule said was worth it. The 0.80 threshold is promoted; had the wrongful-removal rate breached its guardrail, it would not have been.

How it works

What separates this from a hunch or a before/after look is that champion and challenger run concurrently on the same cases, so period effects and population shifts cancel. The promotion metric and the guardrail metrics are fixed before the test, and the decision rule — how much the challenger must win by, over how many cases — is pre-committed, so the result can't be reverse-engineered. Challengers usually run in pure shadow (logged, not enacted) or on a small live slice; either way the champion remains the system of record until a challenger earns its place.

Tuning parameters

  • Shadow vs live split — whether challengers only log or actually decide a small fraction of traffic; live exposure is realer but carries risk.
  • Test duration / sample size — how many cases before a verdict; too short and noise wins, too long and a bad champion persists.
  • Promotion criterion + margin — how large and how consistent the challenger's edge must be to win.
  • Guardrail metrics — the secondary limits (wrongful-action rate, subgroup error) a challenger may not breach even while winning overall.
  • Number of challengers — one clean rival or a slate; more explores the space but invites cherry-picking the luckiest.

When it helps, and when it misleads

Its strength is causal cleanliness: by holding inputs identical it answers "is this boundary actually better?" in a way a naïve before/after never can, and it lets a team explore a change with the incumbent still safely in charge. Its classic misuse is peeking — watching the running comparison and stopping the moment a challenger is briefly ahead, which manufactures a winner out of noise.[1] With several challengers, the same trap becomes cherry-picking the luckiest. The discipline is to pre-register the metric, margin, and stopping rule, and to honor them.

How it implements the components

  • validation_and_backtesting_plan — the shadow/parallel comparison is the validation design: testing a candidate against reality before adoption.
  • recalibration_rule — the pre-committed promotion rule is exactly the rule that converts test evidence into a decision to move the boundary.
  • threshold_performance_metric — the promotion metric and guardrails are the performance numbers the two thresholds are judged on.

It does not deploy the winner gradually across the fleet (rollout_and_guardrail_plan) — Staged Threshold Rollout does — nor does it record the adopted version (threshold_version_registry), which Threshold Versioning Register owns.

Notes

Champion / Challenger decides whether and to what value a threshold should change; Staged Threshold Rollout decides how to deploy the winner safely. A test that promotes a threshold is not the same as a rollout that ships it — running them together (validate, then stage) is the intended sequence, and collapsing the two is how an under-tested threshold reaches production.

References

[1] Repeatedly checking a running comparison and stopping as soon as it looks favorable inflates the false-positive rate — the multiple-comparisons / optional-stopping problem. Pre-registering the metric, effect-size margin, and stopping rule is the standard guard. ↩