Dataset Shift in Machine Learning¶
Quiñonero-Candela, J., Sugiyama, M., Schwaighofer, A., & Lawrence, N. D. (2009). Dataset Shift in Machine Learning. MIT Press.
Cited by¶
7 citations across 7 artifacts.
Each citation links to the sentence it supports in the citing article.
Primes¶
- Calibrated Rule versus Moving World
- The model's deployed performance is its expected loss under the live distribution $P_{\text{live}}(x, y)$, and the non-stationarity invariant is that $P_{\text{live}} \neq P_{\text{train}}$ and moves over time.
This sourceStandard reference formalizing covariate shift and concept shift as P(x) and P(y|x) moving between training and deployment.
- The model's deployed performance is its expected loss under the live distribution $P_{\text{live}}(x, y)$, and the non-stationarity invariant is that $P_{\text{live}} \neq P_{\text{train}}$ and moves over time.
- Foreseeing (Prediction)
- The model was calibrated on historical data but faced a distributional shift
This sourceStandard reference on dataset/covariate shift — when training and test (deployment) distributions differ — and methods to address it; supports the claim that the 2016 election model was calibrated on historical data but faced a distributional shift.
- The model was calibrated on historical data but faced a distributional shift
- Portable Context Bundle
- In trained machine-learning models, weights encode the training distribution, and deployment in a new context applies the model's implicit captured environment — distribution shift is the failure mode the bundle predicts.
This sourceFoundational treatment of distribution/dataset shift, where a model trained on one distribution is deployed on another.
- In trained machine-learning models, weights encode the training distribution, and deployment in a new context applies the model's implicit captured environment — distribution shift is the failure mode the bundle predicts.
- Regime Change
- Neural networks and machine learning: Mode collapse during training (generator captures only a few modes of the data distribution); phase transitions in learning dynamics (ordered to chaotic phases); regime brittleness (models trained on one data regime fail catastrophically when distribution shifts), a failure mode Goodfellow et al. (2014) describe in their original GAN paper and that Quiñonero-Candela et al. (2009) frame more generally as dataset shift.
This sourceCanonical reference on regime brittleness in machine learning: formalizes covariate shift, prior shift, and concept drift as failure modes when models trained on one data regime are deployed in another.
- Neural networks and machine learning: Mode collapse during training (generator captures only a few modes of the data distribution); phase transitions in learning dynamics (ordered to chaotic phases); regime brittleness (models trained on one data regime fail catastrophically when distribution shifts), a failure mode Goodfellow et al. (2014) describe in their original GAN paper and that Quiñonero-Candela et al. (2009) frame more generally as dataset shift.
- Transferability Overclaim
- In machine learning it is the model evaluated on a benchmark and deployed against inputs outside its training distribution, where the benchmark's covariate range silently becomes the deployment's risk boundary.
This sourceStandard treatment of distribution/dataset shift between training benchmark and deployment inputs.
- In machine learning it is the model evaluated on a benchmark and deployed against inputs outside its training distribution, where the benchmark's covariate range silently becomes the deployment's risk boundary.
- Vantage-Induced Omission
- Machine-learning training-data collection has a vantage (web crawl, panel survey, sensor deployment), and the model is blind to regions unreachable from it, producing the distribution-shift failure family.
This sourceA model's training-data collection has a vantage; regions unreachable from it produce the distribution-shift failure family.
- Machine-learning training-data collection has a vantage (web crawl, panel survey, sensor deployment), and the model is blind to regions unreachable from it, producing the distribution-shift failure family.
Mechanisms¶
- Problem Distribution Profile
- It captures the distribution someone expects, which can quietly diverge from the one that actually arrives — the standing risk of dataset shift
This sourceDefines dataset shift as a difference between training and test data distributions.
- It captures the distribution someone expects, which can quietly diverge from the one that actually arrives — the standing risk of dataset shift
Verification¶
This reference passed the adversarial substantiation pipeline: it was checked to exist and to support the claim it is attached to. See how references were verified.
Links previously used in the corpus¶
Before the registry existed this work was also linked 1 other way.
Registry ID ref:7e9f6e158fde · see in the full table