Dataset Shift in Machine Learning¶
Quiñonero-Candela, J., Sugiyama, Masashi, Schwaighofer, Anton, & Lawrence. (2009). Dataset Shift in Machine Learning. The MIT Press.
Cited by¶
14 citations across 14 artifacts.
Each citation links to the sentence it supports in the citing article.
Primes¶
- Calibrated Rule versus Moving World
- The model's deployed performance is its expected loss under the live distribution \(P_{\text{live}}(x, y)\), and the non-stationarity invariant is that \(P_{\text{live}} \neq P_{\text{train}}\) and moves over time.
This sourceStandard reference formalizing covariate shift and concept shift as P(x) and P(y|x) moving between training and deployment.
Supported in partVerified against the publisher's abstract
The abstract confirms dataset shift as training and test distributions differing, backing the P_live ≠ P_train clause, but not the expected-loss formulation or drift over time.
“Dataset shift is a common problem in predictive modeling that occurs when the joint distribution of inputs and outputs differs between training and test stages. Covariate shift, a particular case of dataset shift, occurs when only the input distribution changes.”
- The model's deployed performance is its expected loss under the live distribution \(P_{\text{live}}(x, y)\), and the non-stationarity invariant is that \(P_{\text{live}} \neq P_{\text{train}}\) and moves over time.
- Foreseeing (Prediction)
- The model was calibrated on historical data but faced a distributional shift
This sourceStandard reference on dataset/covariate shift — when training and test (deployment) distributions differ — and methods to address it; supports the claim that the 2016 election model was calibrated on historical data but faced a distributional shift.
- The model was calibrated on historical data but faced a distributional shift
- Portable Context Bundle
- In trained machine-learning models, weights encode the training distribution, and deployment in a new context applies the model's implicit captured environment — distribution shift is the failure mode the bundle predicts.
This sourceFoundational treatment of distribution/dataset shift, where a model trained on one distribution is deployed on another.
Supported in partVerified against the publisher's abstract
The volume's own abstract establishes that dataset/distribution shift is a real and common failure when training and deployment distributions differ, which is the frame of the claim but not its mechanism clause.
“Dataset shift is a common problem in predictive modeling that occurs when the joint distribution of inputs and outputs differs between training and test stages. Covariate shift, a particular case of dataset shift, occurs when only the input distribution changes.”
- In trained machine-learning models, weights encode the training distribution, and deployment in a new context applies the model's implicit captured environment — distribution shift is the failure mode the bundle predicts.
- Regime Change
- Neural networks and machine learning: Mode collapse during training (generator captures only a few modes of the data distribution); phase transitions in learning dynamics (ordered to chaotic phases); regime brittleness (models trained on one data regime fail catastrophically when distribution shifts), a failure mode Goodfellow et al. (2014) describe in their original GAN paper and that Quiñonero-Candela et al. (2009) frame more generally as dataset shift.
This sourceCanonical reference on regime brittleness in machine learning: formalizes covariate shift, prior shift, and concept drift as failure modes when models trained on one data regime are deployed in another.
Supported in partVerified against the publisher's abstract
The book's own abstract frames dataset shift as a general failure when training and test distributions differ, supporting the 'frame more generally' clause but not 'fail catastrophically'.
“Dataset shift is a common problem in predictive modeling that occurs when the joint distribution of inputs and outputs differs between training and test stages. Covariate shift, a particular case of dataset shift, occurs when only the input distribution changes.”
- Neural networks and machine learning: Mode collapse during training (generator captures only a few modes of the data distribution); phase transitions in learning dynamics (ordered to chaotic phases); regime brittleness (models trained on one data regime fail catastrophically when distribution shifts), a failure mode Goodfellow et al. (2014) describe in their original GAN paper and that Quiñonero-Candela et al. (2009) frame more generally as dataset shift.
- Transferability Overclaim
- In machine learning it is the model evaluated on a benchmark and deployed against inputs outside its training distribution, where the benchmark's covariate range silently becomes the deployment's risk boundary.
This sourceStandard treatment of distribution/dataset shift between training benchmark and deployment inputs.
Supported in partVerified against the publisher's abstract
The volume's abstract names covariate shift — input distribution differing between training and test stages — as a real problem, which backs the claim's frame but not its 'risk boundary' clause.
“Covariate shift, a particular case of dataset shift, occurs when only the input distribution changes.”
- In machine learning it is the model evaluated on a benchmark and deployed against inputs outside its training distribution, where the benchmark's covariate range silently becomes the deployment's risk boundary.
- Vantage-Induced Omission
- Machine-learning training-data collection has a vantage (web crawl, panel survey, sensor deployment), and the model is blind to regions unreachable from it, producing the distribution-shift failure family.
This sourceA model's training-data collection has a vantage; regions unreachable from it produce the distribution-shift failure family.
Supported in partVerified against the publisher's abstract
The volume's abstract supports the general frame that collection conditions (experimental-design bias) produce dataset/distribution shift, but not the 'vantage' mechanism or the incident-reporting clause.
“Dataset shift is present in most practical applications, for reasons ranging from the bias introduced by experimental design to the irreproducibility of the testing conditions at training time.”
- Machine-learning training-data collection has a vantage (web crawl, panel survey, sensor deployment), and the model is blind to regions unreachable from it, producing the distribution-shift failure family.
Domain-specific¶
- Gold-Standard Erosion
- The three demand different interventions — retraining, importance weighting, and relabeling respectively — and misdiagnosing one as another misdirects the response.
This sourceSets out the dataset-shift taxonomy and shows that each member - covariate shift, concept or conditional shift, prior-probability shift - calls for a different correction.
- The three demand different interventions — retraining, importance weighting, and relabeling respectively — and misdiagnosing one as another misdirects the response.
Mechanisms¶
- Benchmark Dataset
- Its failure mode is distribution shift: a frozen benchmark slowly diverges from the live task, so a rising score can reflect overfitting to the benchmark rather than real improvement, and a popular set inherits its original blind spots forever.
This sourceDefines dataset shift as divergence between development data and the distribution encountered in evaluation or deployment.
- Its failure mode is distribution shift: a frozen benchmark slowly diverges from the live task, so a rising score can reflect overfitting to the benchmark rather than real improvement, and a popular set inherits its original blind spots forever.
- Deployment Canary and Drift Sentinel
- The underlying phenomenon it hunts is distribution shift — deployment data drifting away from the data the model was trained and validated on.
This sourceDefines dataset or covariate shift as deployment or test data following a distribution different from the data on which a model was trained and evaluated.
- The underlying phenomenon it hunts is distribution shift — deployment data drifting away from the data the model was trained and validated on.
- Image-Region Segmentation Pipeline
- Its central failure mode is that it inherits the distribution it learned on: contours degrade under domain shift, when the deployment images differ from the training ones (a new scanner, a different population, unfamiliar lighting), and the pipeline often reports its old confidence anyway.
This sourceExplains that models inherit training-distribution relationships and can degrade when deployment data come from a shifted domain.
- Its central failure mode is that it inherits the distribution it learned on: contours degrade under domain shift, when the deployment images differ from the training ones (a new scanner, a different population, unfamiliar lighting), and the pipeline often reports its old confidence anyway.
- Lab-to-Field Translation
- The honest anchor here is distribution shift: the statistical relationship the lab measured need not hold once the input distribution changes under live conditions.
This sourceDefines distribution shift as a change from training to deployment conditions under which learned statistical relationships may fail.
- The honest anchor here is distribution shift: the statistical relationship the lab measured need not hold once the input distribution changes under live conditions.
- Model Applicability Card
- Its strength is that it makes a model's domain of validity portable and legible, catching the most dangerous error in applied ML — using a model outside the distribution it was validated on, where its numbers look identical but mean nothing.
This sourceDefines dataset shift as a change in the joint distribution of inputs and outputs between training and test or deployment settings, which can cause a model built under one distribution to fail under another.
- Its strength is that it makes a model's domain of validity portable and legible, catching the most dangerous error in applied ML — using a model outside the distribution it was validated on, where its numbers look identical but mean nothing.
- Problem Distribution Profile
- It captures the distribution someone expects, which can quietly diverge from the one that actually arrives — the standing risk of dataset shift
This sourceDefines dataset shift as a difference between training and test data distributions.
- It captures the distribution someone expects, which can quietly diverge from the one that actually arrives — the standing risk of dataset shift
- Source Attribution Training Set
- If the set stops resembling live use, a judgment that scores well on it can fail in the field: distribution shift quietly turns yesterday's representative corpus into a misleading one.
This sourceDefines dataset shift as training and test distributions differing and shows how that mismatch can make deployed prediction fail.
- If the set stops resembling live use, a judgment that scores well on it can fail in the field: distribution shift quietly turns yesterday's representative corpus into a misleading one.
Verification¶
Does it exist? Confirmed. This work's DOI resolves to a registered record, which fixes its identity. That is all it fixes.
Does it back the claim? Read against the text for 5 of 14 citations: 5 supported in part. Each verdict is shown under its citation below, with what in the work backs the sentence.
Support is checked per citation rather than per work — the same source can be cited soundly in one article and wrongly in another. Per-citation recording began recently, so a citation with no recorded check is a gap in the record rather than evidence it went unchecked.
See how references were verified.
Links previously used in the corpus¶
Before the registry existed this work was also linked 2 other ways.
- https://doi.org/10.7551/mitpress/9780262170055.001.0001 ×10
- https://direct.mit.edu/books/edited-volume/3841/Dataset-Shift-in-Machine-Learning ×1
Registry ID ref:7e9f6e158fde · see in the full table