Residual Comparison Test¶

Residual diagnostic test — instantiates Predictive Residual Processing

Interrogates the shape of the leftover residuals — against a null, a rival model, or a raw sample — to tell honest noise from a model that is quietly wrong.

Suppressing residuals is only safe if what's left after prediction really is unpredictable noise. Residual Comparison Test treats that as a testable hypothesis: it examines the structure of the residual stream and compares it against a reference — a white-noise null, a rival model's residuals, or an occasional full-state sample — to decide whether the leftovers are noise (fine to discard) or patterned signal the model is systematically missing (misspecification). Where a drift monitor asks whether error is growing over time and an anomaly detector asks whether a single point is odd, this asks a structural question: is there a pattern in what we're throwing away? A model can have small, stable residuals that are nonetheless full of structure — and that is exactly the case where suppression quietly discards real information.

Example¶

An inline temperature sensor on a process line is cheap but biased, so it's modeled to predict what an expensive reference probe would read; only the residual gets logged. Periodically the test runs. It computes the residuals and tests them for structure — are they autocorrelated? do they trend with temperature? — and compares them against the residuals of an alternative polynomial model and against a handful of paired reference readings. The white-noise null is rejected: the residuals are autocorrelated and slope upward above ≈80°C. The cheap sensor's model isn't merely imprecise, it's biased at the high end. Verdict: suppression is unsafe in the hot regime; keep raw readings there until the model is refit.

How it works¶

Compute the residuals and test for structure — autocorrelation, trend, non-zero mean, changing variance. White residuals mean the model has extracted the signal; structured residuals mean something predictable was left behind.
Compare against a reference. A rival model's residuals localize which model is missing what; a raw/full-state sample supplies ground truth the residual stream alone can't provide.
Localize the misspecification — the regime, segment, or range where structure appears — so the fix is targeted rather than a blanket retrain.

Tuning parameters¶

Structure tests used — autocorrelation vs. trend vs. heteroscedasticity; each catches a different failure shape.
Comparison reference — null, rival model, or raw sample; richer references detect subtler misspecification at higher cost.
Significance threshold — how strong the evidence of structure must be before you act; loose invites false alarms, tight misses real bias.
Segment vs. pooled — testing within regimes catches localized misspecification a pooled test averages away.
Raw-sampling spend — how much independent ground truth you buy to compare against.

When it helps, and when it misleads¶

Its strength is separating a model that is merely imprecise (noise — safe to suppress) from one that is wrong in a pattern (misspecified — unsafe), which magnitude alone cannot distinguish.

Its failure modes: weak tests miss real structure; running many tests until one is "significant" manufactures spurious structure; and the whole thing is only as trustworthy as the independence of the raw sample it compares against.^[1] The classic misuse is declaring residuals "just noise" without ever comparing them to a rival or to raw truth — assuming precisely what the test exists to check. The discipline: always test against at least one alternative and some independent ground truth, and correct for multiple comparisons.

How it implements the components¶

cross_model_disagreement_monitor — it pits the model's residuals against a rival model's, treating systematic disagreement as evidence of misspecification.
noise_filter — its core act is separating structured residual (missed signal) from genuine noise, so only the latter is safely suppressed.

It does not raise the live degradation signal (that's Model Drift Monitoring), obtain the raw ground-truth sample it compares against (that's Shadow Raw-Channel Sampling), or validate the model across full history before launch (that's Forecast Backtesting).

Instantiates: Predictive Residual Processing — the diagnostic that decides whether the leftovers are safe to discard.
Consumes: Shadow Raw-Channel Sampling — supplies the independent full-state sample it tests residuals against.
Sibling mechanisms: Model Drift Monitoring · Anomaly Detection Model · Forecast Backtesting · Prediction Error Review · Shadow Raw-Channel Sampling

Notes¶

The test is only as good as the independence of its reference. A raw sample drawn from the same corrupted channel, or a rival model sharing the same blind spot, will agree with the suspect model and wrongly certify the residuals as noise — so the value lives in the independence of the comparison, not the comparison itself.

References¶

[1] Structured (e.g., autocorrelated) residuals as a signature of model misspecification is a standard diagnostic idea — tests such as Durbin–Watson (autocorrelation) and Ljung–Box (residual independence) formalize "are the leftovers really noise?" Correcting for multiple comparisons guards against manufacturing structure by repeated testing. ↩