Rolling Window Comparison¶

Comparative diagnostic — instantiates Moving-Target Tracking

Quantifies how much the target, state, and error distributions have drifted by comparing a recent window against earlier ones — turning gradual staleness into a measured magnitude rather than a yes/no event.

Where a change-point detector asks "did the world just break?", Rolling Window Comparison asks the slower, quantitative question: "how far has it drifted since?" It slides a window over the recent target, state, and error streams and compares that window's distribution against one or more earlier windows, producing a magnitude of drift and staleness — how much the mean has moved, how the spread has changed, how stale the current model's assumptions have become. Its defining feature is that its output is a measured quantity, not an event: not "a break occurred" but "the demand distribution has shifted this far and the loop's error has grown this much over the last N periods." That makes it the natural complement to abrupt-break detection — it catches the slow, monotone drift that never presents a sharp discontinuity and would slip past an event detector entirely.

Example¶

A retailer's replenishment system was tuned to a demand pattern that no single day ever violates dramatically — but over a season, tastes shift. Rolling Window Comparison slides a trailing four-week window across each SKU's sales and forecast-error streams and compares it against the trailing year: for a category of products it finds the recent demand distribution has drifted up 12% in the mean and widened in variance, and that forecast error has grown steadily rather than in any single jump. No day looked anomalous; the comparison across windows is what surfaces the creep.

The output is a ranked drift estimate — these categories have drifted materially, these are stable, this is how stale each forecast now is — handed to the team that decides whether to re-fit the demand model or adjust buying. Because it reports a magnitude, it also supports a threshold: drift past a set size is what escalates a SKU for a refresh, while smaller drift is logged and watched. It measures the creep that a break detector, tuned for jumps, would never have flagged.

How it works¶

What distinguishes it from an event detector is that it measures a distance between distributions rather than firing on a break:

It maintains a recent window and one or more reference windows, and computes a divergence or distributional distance between them (a stability index, a two-sample statistic, a shift in moments).
It reports a continuous magnitude — how far the target has drifted and how much the tracking error has grown — not a timestamped event.
It attaches a staleness estimate: given the measured drift and the loop's update rate, how out-of-date the current model or plan now is.
It is deliberately window-length aware: the same drift looks larger or smaller depending on how long a window smooths it, so the window choice is part of the reading.

Tuning parameters¶

Window length — how much history each window spans. Short windows react quickly but are noisy and unstable; long windows are smooth and confident but slow to register real drift. The master trade-off.
Reference choice — what "before" means: a fixed baseline period, a lagged window, or a trailing long-run average. A fixed baseline measures total drift from a known-good state; a lagged window measures recent rate of change.
Distance metric — how distribution shift is scored (mean shift, variance change, a population-stability-style divergence, a rank test); different metrics are sensitive to different kinds of drift.
Drift threshold — how large a measured shift counts as material enough to escalate. Lower catches creep early but escalates often; higher waits for unambiguous drift.
Slice granularity — whether comparison runs on the aggregate or per segment/SKU; finer slices catch localized drift the aggregate averages away, at more compute and multiple-comparison risk.

When it helps, and when it misleads¶

Its strength is measuring the gradual — the slow creep that averages out of any single reading but compounds across a season, and that an event detector built for jumps will miss by design. By reporting a magnitude and a staleness, it lets the loop decide how much re-fitting or adjustment the drift actually warrants, proportionate to the size of the shift.

Its failure modes turn on the window and the multiplicity. Too short a window and normal noise reads as drift, prompting needless churn; too long and real drift is smoothed until it is discovered late — the same concept-drift blind spot the mechanism exists to close, reintroduced through a lazy window choice.^[1] Slicing finely across many segments invites false positives from multiple comparisons — some window will always look drifted somewhere. And its classic misuse is choosing the window boundaries after seeing the data so the comparison shows the drift the analyst expected. The discipline that guards against this is to fix window lengths and thresholds in advance, correct for the number of slices tested, and pair the magnitude with an independent break detector so gradual and abrupt change are read on their own terms.

How it implements the components¶

Rolling Window Comparison fills the archetype's drift-measurement slot — the parts that quantify how far things have moved, not the parts that detect a break or fix it:

tracking_error_metric — it computes and compares the error distribution across windows, producing the measured error and how it has grown.
target_trajectory_and_drift_estimator — the window-to-window comparison is the drift estimate: how far and how fast the target's distribution has moved.

It does not fire on abrupt regime breaks (that discrete, timestamped detection is Change-Point Detection), maintain a live operational health monitor (Model Drift Monitoring), or act on the drift by re-fitting the model (Model Retuning). It measures the magnitude and hands it on.

Instantiates: Moving-Target Tracking — Rolling Window Comparison supplies the measured drift-and-staleness the loop uses to decide when a refresh is warranted.
Consumes: State-Estimation Filter — the state and error streams it windows over are produced by the estimator.
Sibling mechanisms: Change-Point Detection · Model Drift Monitoring · Model Retuning · Rolling Forecast Review · State-Estimation Filter

Notes¶

It is the gradual-drift half of a detection pair: Change-Point Detection catches sharp breaks as events, Rolling Window Comparison measures slow creep as a magnitude. Neither subsumes the other — a slow drift never trips the break detector, and a single sharp jump is smeared by the windowing — so a robust loop usually runs both. Note too that it diagnoses without prescribing: a measured drift is an input to the decision to re-fit or adjust, not the re-fit itself.

References¶

[1] Concept drift — the gradual change over time in the underlying relationship a model or plan assumes, so that a once-accurate model silently decays. Rolling Window Comparison exists to measure it; choosing too long a window quietly reintroduces the very blindness it is meant to remove. ↩