Skip to content

Drift Sample Review

Maintenance monitor — instantiates Prototype-Centered Category Modeling

Periodically re-judges a fresh sample of recent cases to catch the category's prototype drifting, and triggers revision of the reference cases before the drift is baked in.

Categories move. What counted as a central example last year sits at the rim today, because the world the category describes keeps changing. A Drift Sample Review is the recurring check that catches that motion. Its distinguishing feature is that it is longitudinal: it pulls a fresh sample of recent cases each period, re-judges them against the current prototype, and watches for the prototype and the world sliding apart. Where an audit compares sources at one moment, this compares the category against itself over time — and when the gap grows, it fires a revision signal and refreshes the reference cases the rest of the system relies on.

Example

A mail provider's "phishing" category is a moving target — attackers change tactics faster than any definition. Each month the trust-and-safety team samples a few hundred recently classified messages and re-judges them by hand against the current prototype. For three quarters, agreement holds. Then the sample turns up a run of messages that are plainly phishing to a human but score low against the prototype: a new style — a fake internal HR notice rather than the old fake-bank pattern — that the reference cases simply do not cover. That is the drift signal. The review retrains nothing itself; it flags that the prototype has fallen behind and adds the new exemplars to the reference-case library, so the next calibration and the classifier are working from a "phishing" that matches this month's reality rather than last year's.

How it works

The distinguishing mechanic is the repeated fresh sample. Each cycle draws recent cases (weighted toward the marginal band, where drift shows first), re-judges them against the standing prototype, and compares the result to prior cycles. A stable gap is noise; a growing gap — cases the world now treats as central that the prototype scores as marginal, or the reverse — is the drift signal. On that signal the review updates the reference-case library with the newly representative examples, so the drift is corrected at its source rather than papered over downstream. It is a monitor: its job is to notice and trigger, not to reclassify everything.

Tuning parameters

  • Review cadence — how often to sample; frequent reviews catch drift early but cost steady effort, rare ones risk letting the prototype rot between checks.
  • Sample size and weighting — larger, marginal-weighted samples spot subtle drift sooner; small random ones are cheap but slow to see it.
  • Drift threshold — how big a gap must grow before it counts as drift rather than noise — the dial between chasing every fluctuation and missing a real shift.
  • Baseline window — compared against last cycle, or against a fixed original baseline; a moving baseline can normalise slow drift into invisibility.
  • Response coupling — whether a drift signal auto-updates the reference library or merely proposes changes for a human to ratify.

When it helps, and when it misleads

Its strength is catching slow change before it becomes silent failure — the case where a classifier or a team keeps applying yesterday's prototype to a category that has quietly moved, and accuracy erodes without any single obvious error.[1] By feeding fresh exemplars back into the reference library, it keeps the whole apparatus current instead of letting it calcify.

It misleads when it mistakes noise for drift and chases every wobble, churning the prototype on random variation; or the reverse, when a slowly moving baseline lets genuine drift hide because each cycle looks like the last. Over-eager revision can also erase a stable category's identity by rewriting it toward whatever this month's sample happened to contain. The discipline is to require a sustained gap before revising, hold at least one fixed baseline so slow drift cannot creep past, and keep human ratification on changes to the reference cases.

How it implements the components

The review fills the maintenance-side components a monitor can own:

  • prototype_drift_and_revision_signal — its core output: the flag that the prototype and the world have moved apart far enough to warrant revision.
  • calibration_case_library — on a drift signal it refreshes the reference cases with newly representative examples, keeping the library current.

It monitors and triggers but does not build the library in the first place — that's a Calibration Workshop — and it detects change over time rather than disagreement at a moment, which is a Classification Disagreement Audit.

References

[1] Concept drift — the phenomenon, well known in machine learning, where the statistical relationship a model was built on changes over time, so a once-accurate category silently decays. Periodic re-sampling against fresh data is the standard way to detect it. ↩