Skip to content

Change-Point Detection

Detection monitor — instantiates Moving-Target Tracking

Flags the moment the target jumps to a new regime — an abrupt discontinuity the current tracking mode can no longer follow — so the loop switches modes instead of chasing a break as if it were noise.

Most tracking machinery assumes the target moves smoothly and asks "how far off are we?" Change-Point Detection asks a different, binary question: has the world just switched regimes? It watches the target and error streams for an abrupt discontinuity — a step, a slope reversal, a jump in variance — and, when it finds one, emits an event with a timestamp rather than a distance. That event is the trigger the rest of the loop hangs a mode switch on: the smooth-tracking assumptions that held a minute ago are now void, and continuing to filter, predict, and correct against the old regime would drag the estimate away from the truth. Its defining move is treating a break as a discrete thing to be detected — not a large error to be reduced.

Example

A platform's abuse classifier has tracked a slowly evolving spam population for months; its false-negative rate drifts gently and the team retunes on a monthly cadence. One night a coordinated ring switches tactics wholesale — new account-creation pattern, new payload, new timing. A gradual drift metric would take weeks to notice, because it averages the break into the trailing baseline. Change-Point Detection, running a sequential test on the appeal rate and score distribution, instead flags a break at 02:14 UTC: the generating distribution has shifted, sharply, and the flag fires within hours, not weeks.

The output is not "error is up 3%" but "a regime change occurred at 02:14; the current mode is no longer valid." That single event routes the system into a stricter provisional mode and queues human review — a response the smooth loop would never have initiated, because to a drift metric nothing had gone dramatically wrong yet.

How it works

What distinguishes it from ordinary error monitoring is that it tests for a break in the data-generating process, not a level of error:

  • It runs a sequential test — CUSUM, a Bayesian online change-point method, a likelihood-ratio scan — that accumulates evidence that the recent segment came from a different distribution than the reference segment.
  • It emits a discrete event with a time index, not a continuous magnitude, and declares which signals broke.
  • It deliberately trades detection delay against false-alarm rate: raise the evidence threshold and you miss less noise but react later; lower it and you react fast but cry wolf.
  • Its adversarial variant assumes the break may be deliberate — an actor probing then jumping — so it watches for the signatures of gamed, engineered discontinuities, not just natural ones.

Tuning parameters

  • Detection threshold — how much accumulated evidence before it fires. Higher suppresses false alarms but lengthens detection delay; lower reacts sooner but chatters. This is the master dial.
  • Reference window — what counts as "before." A short reference adapts to slow drift but can mistake it for a break; a long reference is stable but sluggish to reset after a real switch.
  • Minimum run length — how many post-break observations required before the change is confirmed, filtering one-off spikes from true regime shifts.
  • Signal breadth — a single series or a multivariate break test across many signals at once; broader catches subtler joint shifts but raises false-alarm risk.
  • Retrospective vs. online — scan history for where breaks were (clean, unhurried) versus a live sequential monitor that fires as they happen (timely, noisier).

When it helps, and when it misleads

Its strength is catching the discontinuities that averaged drift metrics smear into invisibility — the overnight regime switch, the structural break, the moment an old model stops applying at all. It is the sensor that lets a tracking loop change modes rather than grind forward on stale assumptions.

Its failure modes are the mirror image of its tuning. Too sensitive and it drowns operators in false alarms until they stop looking — genuine alarm fatigue, at which point a real break is missed for a human reason rather than a statistical one.[1] Too conservative and it detects the break long after the damage is done. Its classic misuse is running it backwards: tuning the threshold or picking the window after glancing at the data so the test "detects" a break the analyst already believed in — a break point can almost always be manufactured post hoc. The discipline that guards against this is to fix the alarm rule before the data arrives and to pair every fired event with an independent magnitude estimate before acting on it.

How it implements the components

Change-Point Detection fills the archetype's discontinuity-detection slot — the parts that recognize a regime break and route the switch, not the parts that measure or respond to it:

  • tracking_mode_selection_rule — a detected break is precisely the trigger that selects a new tracking mode; the detector supplies the signal on which the mode rule keys.
  • adversarial_target_change_monitor — its adversarial variant is the monitor for deliberate, engineered jumps, watching for the fingerprints of gamed target moves rather than natural drift.

It does not measure how far the target has gradually drifted (tracking_error_metric, target_trajectory_and_drift_estimator — those are Rolling Window Comparison), nor does it re-fit the model or retune the loop in response; the standing health monitor is Model Drift Monitoring's job. It only detects the break and names the moment.

  • Instantiates: Moving-Target Tracking — Change-Point Detection supplies the "regime just switched" event the loop's mode-selection hangs on.
  • Consumes: State-Estimation Filter — the residual and error streams it watches for a break are produced by the estimator.
  • Sibling mechanisms: Rolling Window Comparison · Model Drift Monitoring · Policy Recalibration · Rolling Forecast Review · State-Estimation Filter

Notes

A change point is necessary but not sufficient to switch modes — the detector says "something broke," not "here is the new regime." Pair it with a magnitude estimator (Rolling Window Comparison) to size the break and with a response mechanism to act on it; used alone it can only raise the alarm. It is also biased toward abrupt change: a slow, monotone drift that never presents a sharp break will slip past it entirely, which is exactly why the two detectors are complementary rather than redundant.

References

[1] Alarm fatigue — the desensitization that sets in when a monitor fires too often, so that operators begin ignoring or muting it and eventually miss the true positive buried among false ones. It is why the detection-threshold dial is a human-factors decision as much as a statistical one. ↩