Annotation Drift¶

The gradual, undocumented shift over time in how annotators apply an unchanged rubric, so identical instances get different labels — a silent recalibration of a measurement instrument, invisible to within-slice agreement and detectable only by re-annotating frozen gold.

Core Idea¶

Annotation drift is the gradual, undocumented change over time in how human annotators apply a labeling rubric, so nominally identical instances at different times receive different labels — not because the phenomenon changed, not because the rubric was revised, but because the annotators' shared interpretive practice silently shifted. It is a calibration problem for a non-stationary measurement instrument, distinct from concept drift (the world changed) and annotation error (noise around a stable instrument) by its time-varying, cohort-level, within-rubric character.

Scope of Application¶

Annotation drift lives across the labeling and evaluation subfields of ML and data science — wherever a longitudinal label stream is produced by a human-applied rubric whose interpretive calibration can silently slide while the document stays fixed.

Clinical-prediction model training — chart abstractors' ICD and outcome conventions sliding over years.
Content-moderation labeling — the same post labeled differently across years under unchanged policy.
Search-relevance rating — raters drifting toward newer notions of a "good" result.
Crowdsourced computer-vision labeling — definitions sliding with new guidelines and annotator pools.
Sentiment / toxicity labeling — shifting cultural baselines changing what counts as toxic.

Clarity¶

Naming annotation drift separates two things identical on a falling validation curve: real concept drift in the world and measurement drift in the labeling instrument. It forces a prior instrumentation question — is our rubric calibrated the same across time? — and explains why the usual QA instrument gives no warning: within-slice inter-annotator agreement stays high while the whole cohort drifts together.

Manages Complexity¶

The pathologies are an unruly list, each with its own apparent cause, hiding on a falling curve among rival explanations. The concept compresses them under one framing — the rubric is a measurement instrument whose calibration silently shifts in use — so the analyst tracks a single quantity, calibration across time cohorts, and reads diagnosis, boundary classification, and remedy off it.

Abstract Reasoning¶

The reframing licenses diagnostic discrimination (concept versus measurement drift, with retraining-on-recent predicted to bake the artifact in), the non-obvious inference that within-slice agreement is structurally blind while only frozen-gold re-annotation can detect drift, boundary-drawing on three conditions (time-varying not offset, cohort-level not per-instance, within-rubric not revised), and an interventionist family of instrument-stabilizing measures.

Knowledge Transfer¶

Within ML evaluation and labeling the concept transfers as mechanism across subfields sharing one substrate — a longitudinal human-applied label stream — the reframing, diagnostic discrimination, frozen-gold inference, boundary conditions, and intervention family carrying intact. Beyond ML the pattern is a genuine shared mechanism already named elsewhere (coder/rater drift, diagnostic drift, sentencing drift): a stable-on-paper instrument recalibrating in use. It is carried by the parent instrument_interpretive_drift, a distinct neighbor of measurement_uncertainty (noise around a stable instrument); the ML cargo and the name stay home.

Relationships to Other Abstractions¶

Current abstraction Annotation Drift Domain-specific

Parents (1) — more general patterns this builds on

Annotation Drift is a kind of Instrument Interpretive Drift Prime

Annotation drift is instrument-interpretive drift specialized to a human labeling practice whose written rubric remains nominally fixed.

Hierarchy paths (2) — routes to 2 parentless roots

Annotation Drift → Instrument Interpretive Drift → Temporal Decay and Degradation → Entropy (Thermodynamic Sense)

Show alternative path (1)

Neighborhood in Abstraction Space¶

Annotation Drift sits in a sparse region of the domain-specific corpus (87^th percentile for distinctiveness): few abstractions share its structure, so a faithful description tends to retrieve it precisely.

Family — Qualitative Research Rigor & Reflexivity (14 abstractions)

Nearest neighbors

Computed from structural-signature embeddings · 2026-07-12