Drift and Change-Point Detection¶
Detection algorithm — instantiates Proxy–Target Divergence Detection and Recalibration
Watches the proxy's own signal stream for abrupt breaks and gradual drift, flagging when its statistical behavior changes even before anyone measures the target.
Drift and Change-Point Detection is the cheap, always-on tripwire of the fidelity loop. It watches the proxy's own stream — its distribution, its mean, its variance, the inputs feeding it — and raises a flag the moment that stream stops behaving the way it used to. Its defining move is that it never touches the target: it needs no ground truth, no audit, no second channel, only the proxy's history. That is its power and its limit. A change point tells you precisely when the world underneath the proxy shifted; it cannot, by itself, tell you whether the shift broke the proxy's link to the target or was harmless. It converts a silent statistical break into a timestamped, thresholded alarm that the rest of the loop can then investigate.
Example¶
A retail bank runs a machine-learned fraud score on every card transaction. For two years the score's distribution is stable — most transactions cluster near zero, a thin tail near one. Then, over three weeks, the detector notices the input feature "seconds since last transaction" has quietly shifted: its median has halved. A change-point test on that feature crosses its control limit and fires. Nobody has yet confirmed any fraud outcome — the labels take ninety days to settle — but the algorithm has already marked a regime boundary: something about how customers transact changed on or around the 12th. Investigation later finds a new instant-checkout flow shipped that week, compressing inter-transaction times and pushing legitimate transactions into a band the model had learned to treat as suspicious. The detector did not diagnose the cause; it caught the break early enough that the fraud team could hold the model's threshold constant instead of chasing a spike in false declines.
How it works¶
The method is a family of statistical monitors laid over the proxy stream:
- Distributional drift tests compare a recent window of the signal against a reference window — population stability index, KL divergence, or a two-sample test — and score how far the recent shape has moved.
- Change-point tests (CUSUM, Bayesian online change-point, sequential likelihood-ratio[1]) hunt for the specific moment a parameter shifted, rather than merely that it did.
- Input-side vs. output-side monitoring watches both the proxy's output and the features that feed it, since inputs often drift weeks before the output visibly degrades.
Each monitor carries a control limit; crossing it emits a dated sentinel event and stamps a candidate regime boundary. The detector's output is a when and a how much, never a why — that hand-off is deliberate.
Tuning parameters¶
- Window sizes (reference vs. detection) — a long reference window is stable but slow to accept legitimate new normals; a short detection window catches breaks fast but chatters on noise.
- Alarm threshold — how large a statistical excursion trips the sentinel. Tighter thresholds catch subtle drift at the cost of false alarms; the whole trade-off is sensitivity against alarm fatigue.
- Drift-vs-change-point emphasis — whether to prioritize slow-creep detection or abrupt-break localization; most streams need both, weighted by how the proxy tends to fail.
- Feature coverage — monitor the output only, or every upstream input too. Broader coverage gives earlier warning but multiplies the number of things that can cry wolf.
- Persistence filter — how many consecutive breaches before a flag escalates, suppressing one-off blips.
When it helps, and when it misleads¶
Its strength is earliness and cost. Because it runs on data the system already produces, it is the first thing to move when a population shifts, an instrument re-scales, or an upstream pipeline changes — often long before any target-side evidence exists. It is the sentinel that buys the rest of the loop time.
Its central failure mode is that a statistical break is not the same as a fidelity break, and vice versa. The detector fires on harmless shifts (a seasonal pattern, a benign product change) and — more dangerously — stays silent when the proxy decouples without any change in its own distribution, which is exactly what strategic gaming often produces: actors keep the metric's shape identical while hollowing out its meaning.[2] Treating a green drift-monitor as evidence the proxy is still valid is the classic misuse; drift-quiet is not the same as fidelity-confirmed. The guarding discipline is to wire every fired change point into a target-side investigation rather than an automatic action, and never to read the absence of drift as positive proof of fidelity — that proof only comes from measuring the target.
How it implements the components¶
divergence_sentinel— the fired change point is the sentinel event; a statistical breach converts silent decay into a dated, reviewable trigger.regime_and_context_marker— each detected break stamps a boundary in time, partitioning the proxy's history into "before" and "after" regimes downstream analysis can condition on.decoupling_threshold_rule— the control limit is the pre-committed rule that decides, without debate, when an excursion is large enough to count.
This detector never measures the target, so it does not implement independent_target_check or baseline_fidelity_evidence — those belong to Holdout Ground-Truth Audit; nor does it watch downstream harm — that harm_and_consequence_monitor role is Sentinel Outcome Dashboard.
Related¶
- Instantiates: Proxy–Target Divergence Detection and Recalibration — it supplies the early, target-free tripwire the loop's slower checks are triggered by.
- Sibling mechanisms: Holdout Ground-Truth Audit · Incentive Impact Review · Proxy Retirement Decision Record · Proxy–Target Correlation Refresh · Reference-Standard Recalibration Review · Sentinel Outcome Dashboard · Shadow Target Measurement · Triangulated Proxy Panel
Editorial Notes¶
Form Classification¶
Form family: Monitoring, Sensing & Alerting
Rationale: Drift and Change-Point Detection operates as an ongoing sensing arrangement that repeatedly observes actual state and surfaces changes or alerts because it watches the proxy's own signal stream for abrupt breaks and gradual drift, flagging when its statistical behavior changes even before anyone measures the target.
Independent corroboration: The frozen evidence defines Drift and Change-Point Detection as 'Watches the proxy's own signal stream for abrupt breaks and gradual drift, flagging when its statistical behavior changes even before anyone measures the target', so its operative form is Monitoring, Sensing & Alerting.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Statistics & Experimental Design
Origin pattern: Single lineage
Present-day reach: Multi-domain
Rationale: Statistical process control cohered change-point methods such as CUSUM for accumulating small deviations until a persistent shift crosses a decision bound.
Related originating lineages:
- Data Science & Analytics — Model monitoring generalized abrupt and gradual drift detection to features, predictions, and proxies.
Review resolution: Statistical process control and sequential change detection are primary, with model monitoring a genuine data-science extension of the method.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
Drift detection is a trigger, not a verdict. Its worst deployments are the ones that auto-remediate on a fired flag — retraining or re-thresholding a model the instant drift trips — because that closes the loop without ever asking whether the target still moved with the proxy. Keep the alarm and the diagnosis separate.
References¶
[1] Basseville, M., & Nikiforov, I. V. Detection of Abrupt Changes: Theory and Application. Prentice Hall (1993). Develops CUSUM and likelihood-based sequential procedures for detecting abrupt changes and locating the time of a shift. registry ↩
[2] The cumulative-sum (CUSUM) control chart, introduced by E. S. Page in 1954, accumulates small deviations from a reference level so that a persistent shift trips a bound even when no single observation looks anomalous. It is a canonical change-point monitor and illustrates the mechanism's blind spot: it detects shifts in the observed signal, so a decoupling that leaves the signal's distribution intact — the signature of a gamed metric — passes it unseen. withdrawn registry ↩