Leakage in Data Mining¶
Kaufman, S., Rosset, S., Perlich, C., & Stitelman, O. (2012). Leakage in Data Mining: Formulation, Detection, and Avoidance. ACM, 6(4), 1-21.
Cited by¶
17 citations across 17 artifacts.
Each citation links to the sentence it supports in the citing article.
Primes¶
- Data Leakage
- The leak can come from the target itself (the answer is encoded in the inputs), from the future relative to the decision time (information that will not exist when the decision is made leaks back into training), from the test set into the training set (the evaluation is no longer naive), or from the evaluator into the evaluated (the auditor's position is contaminated).
This sourceDefines data leakage in predictive modeling, classifies its channels (target leakage, train-test contamination, leaky features), and prescribes time-based separation by construction.
Supported in partVerified against the publisher's abstract
The abstract backs leakage as illegitimately available target information cured by learn-predict separation, but names none of the future, test-into-train, or evaluator channels the claim lists.
“Deemed “one of the top ten data mining mistakes”, leakage is the introduction of information about the data mining target that should not be legitimately available to mine from.”
- The leak can come from the target itself (the answer is encoded in the inputs), from the future relative to the decision time (information that will not exist when the decision is made leaks back into training), from the test set into the training set (the evaluation is no longer naive), or from the evaluator into the evaluated (the auditor's position is contaminated).
- Feature Engineering
- The second is leakage — using features the downstream consumer would not actually possess at decision time — which recurs across machine learning (train-test contamination), education (items that mirror the lesson too closely), policy (indicators baking in the outcome they predict), and medicine (biomarkers measured downstream of the disease).
This sourceDefines feature/target leakage — using information unavailable at decision time — and methods to detect and avoid it, supporting the leakage hazard in both the failure-mode discussion and the fraud example.
Supported in partVerified against the publisher's abstract
The paper's abstract defines leakage as using information about the data-mining target that is not legitimately available — the machine-learning case of the claim — but says nothing about education, policy, or medicine.
“leakage is the introduction of information about the data mining target that should not be legitimately available to mine from”
- The second is leakage — using features the downstream consumer would not actually possess at decision time — which recurs across machine learning (train-test contamination), education (items that mirror the lesson too closely), policy (indicators baking in the outcome they predict), and medicine (biomarkers measured downstream of the disease).
- Paradox of Unanimity
- A machine-learning team reports a new classifier achieving 100% accuracy on its held-out validation set.
This sourceDefines data leakage and explains how it produces implausibly perfect validation performance.
Supported in partVerified against the publisher's abstract
The abstract defines leakage as target information a model should not legitimately have, which is the mechanism behind implausibly perfect validation results, but it never reports a 100% accuracy classifier.
“leakage is the introduction of information about the data mining target that should not be legitimately available to mine from”
- A machine-learning team reports a new classifier achieving 100% accuracy on its held-out validation set.
- Past-State Contamination
- The artifact may be a reconstruction, an estimate, a fitted model, a decision record, or an evaluation; what makes it a candidate is not its form but its claim, which is that it shows what was, or could have been, held at t.
This sourceFormalises leakage as the entry of information illegitimate with respect to a target, enumerates the channels it arrives through, and prescribes a data-management discipline — learn-predict separation — that closes them without anyone adjudicating the content of what crossed.
Supported in partVerified against the publisher's abstract
Kaufman et al.'s abstract defines leakage as target-relevant information that should not be legitimately available and offers a data-management discipline (learn-predict separation) for avoiding it, but names no specific channel.
“Deemed “one of the top ten data mining mistakes”, leakage is the introduction of information about the data mining target that should not be legitimately available to mine from.”
- The artifact may be a reconstruction, an estimate, a fitted model, a decision record, or an evaluation; what makes it a candidate is not its form but its claim, which is that it shows what was, or could have been, held at t.
Mechanisms¶
- Cross-Validation Weight Calibration
- Data leakage — folds that share information, or look-ahead in time-series splits — inflates measured skill and produces overconfident weights that collapse in deployment.
This sourceDefines data leakage as inappropriate information entering model construction and shows that it produces overly optimistic estimates of predictive performance.
- Data leakage — folds that share information, or look-ahead in time-series splits — inflates measured skill and produces overconfident weights that collapse in deployment.
- Data Leakage Audit
- The failure it targets is classically called target leakage.
This sourceDefines target leakage as introducing information about the prediction target that is not legitimately available to the model.
- The failure it targets is classically called target leakage.
- Exact-or-Numerical Benchmark
- Its failure mode is benchmark leakage
This sourceDefines benchmark leakage as using evaluation information in model or parameter selection, producing an in-sample estimate that overstates out-of-sample performance.
- Its failure mode is benchmark leakage
- Feature Ablation and Holdout Validation
- A weak learner can make a genuinely informative variable look useless, so "failed minimality" is always relative to the model class; and data leakage in the holdout can make an insufficient blanket look sufficient.
This sourceExplains that data leakage contaminating validation can produce overly optimistic estimates of model performance.
- A weak learner can make a genuinely informative variable look useless, so "failed minimality" is always relative to the model class; and data leakage in the holdout can make an insufficient blanket look sufficient.
- Feature Availability Audit
- The guarding discipline is to define the enemy explicitly as target leakage
This sourceDefines target leakage as illegitimate target information and combines formal review with methods for detecting leakage in observed data.
- The guarding discipline is to define the enemy explicitly as target leakage
- Held-Out Sample Test
- Data leakage is the failure mode: any path by which the held-out data influenced the method — shared preprocessing statistics, near-duplicate records across the split, tuning against the test — turns the test into theater.
This sourceDefines data leakage as using target-related information that would not legitimately be available to the deployed model.
- Data leakage is the failure mode: any path by which the held-out data influenced the method — shared preprocessing statistics, near-duplicate records across the split, tuning against the test — turns the test into theater.
- Holdout Leakage Test
- Its failure mode is that it catches only the leakage it is told to look for: temporal leakage, or a confounder like a scanner signature shared across the split, sails past a duplicate scan, and — worse — a passed test breeds false confidence.
This sourceIdentifies temporal leakage as a class of data leakage in predictive modeling.
- Its failure mode is that it catches only the leakage it is told to look for: temporal leakage, or a confounder like a scanner signature shared across the split, sails past a duplicate scan, and — worse — a passed test breeds false confidence.
- Label Proxy Screen
- The Label Proxy Screen is the cheap, standing scan that hunts for exactly this signature — target leakage
This sourceDefines target leakage and presents methodology for detecting and avoiding illegitimate target information in predictive modeling.
- The Label Proxy Screen is the cheap, standing scan that hunts for exactly this signature — target leakage
- Normalization & Scaling Pipeline
- Overfitting Prevention Check
- Its most valuable catch is often data leakage — information that sneaks from the target into the features and inflates measured performance in ways that cannot survive deployment.
This sourceDefines data leakage as illegitimate target information entering predictors and shows how it creates overly optimistic performance that fails in real use.
- Its most valuable catch is often data leakage — information that sneaks from the target into the features and inflates measured performance in ways that cannot survive deployment.
- Rolling Baseline Comparison
- The other classic misuse is leakage: including the current observation (or future data) in the window flatters the comparison and understates real departures.
This sourceDefines data leakage as using information about a prediction target that would not legitimately be available when the prediction is made.
- The other classic misuse is leakage: including the current observation (or future data) in the window flatters the comparison and understates real departures.
- Segmented Holdout Validation
- Its defining failure is data leakage: if the holdout is not truly independent — a shared data source, a leaked target, or a set quietly reused during tuning — it validates nothing and hands back false confidence, often more dangerous than no validation at all.
This sourceDefines leakage as target information illegitimately available to the model and documents severe, misleadingly high evaluation results from such contamination.
- Its defining failure is data leakage: if the holdout is not truly independent — a shared data source, a leaked target, or a set quietly reused during tuning — it validates nothing and hands back false confidence, often more dangerous than no validation at all.
- Train/Test Split
- The failure this exists to catch is leakage — any path, however innocent, by which test information seeps into fitting.
This sourceDefines data leakage as target-related information that is not legitimately available entering model development, including leakage that violates learn–predict separation.
- The failure this exists to catch is leakage — any path, however innocent, by which test information seeps into fitting.
Verification¶
Does it exist? Confirmed. This work's DOI resolves to a registered record, which fixes its identity. That is all it fixes.
Does it back the claim? Read against the text for 4 of 17 citations: 4 supported in part. Each verdict is shown under its citation below, with what in the work backs the sentence.
Support is checked per citation rather than per work — the same source can be cited soundly in one article and wrongly in another. Per-citation recording began recently, so a citation with no recorded check is a gap in the record rather than evidence it went unchecked.
See how references were verified.
Registry ID ref:a09e5a6a7c55 · see in the full table