Leakage in Data Mining

Kaufman, S., Rosset, S., Perlich, C., & Stitelman, O. (2012). Leakage in Data Mining: Formulation, Detection, and Avoidance. ACM Transactions on Knowledge Discovery from Data, 6(4), 1-21.

Type

Journal article

Intellectual base

Primary research

Year

2012

DOI

10.1145/2382577.2382579

Link

https://doi.org/10.1145/2382577.2382579

Cited from

data_science, epistemology, statistics_experimental_design

Cited by

8 citations across 8 artifacts.

Each citation links to the sentence it supports in the citing article.

Primes

• Data Leakage
  • The leak can come from the target itself (the answer is encoded in the inputs), from the future relative to the decision time (information that will not exist when the decision is made leaks back into training), from the test set into the training set (the evaluation is no longer naive), or from the evaluator into the evaluated (the auditor's position is contaminated).

    This sourceDefines data leakage in predictive modeling, classifies its channels (target leakage, train-test contamination, leaky features), and prescribes time-based separation by construction.

• Feature Engineering
  • The second is leakage — using features the downstream consumer would not actually possess at decision time — which recurs across machine learning (train-test contamination), education (items that mirror the lesson too closely), policy (indicators baking in the outcome they predict), and medicine (biomarkers measured downstream of the disease).

    This sourceDefines feature/target leakage — using information unavailable at decision time — and methods to detect and avoid it, supporting the leakage hazard in both the failure-mode discussion and the fraud example.

• Paradox of Unanimity
  • A machine-learning team reports a new classifier achieving 100% accuracy on its held-out validation set.

    This sourceDefines data leakage and explains how it produces implausibly perfect validation performance.

• Past-State Contamination
  • The artifact may be a reconstruction, an estimate, a fitted model, a decision record, or an evaluation; what makes it a candidate is not its form but its claim, which is that it shows what was, or could have been, held at t.

    This sourceFormalises leakage as the entry of information illegitimate with respect to a target, enumerates the channels it arrives through, and prescribes a data-management discipline — learn-predict separation — that closes them without anyone adjudicating the content of what crossed.

Mechanisms

• Data Leakage Audit
  • The failure it targets is classically called target leakage.

    This sourceDefines target leakage as introducing information about the prediction target that is not legitimately available to the model.

• Exact-or-Numerical Benchmark
  • Its failure mode is benchmark leakage

    This sourceDefines benchmark leakage as using evaluation information in model or parameter selection, producing an in-sample estimate that overstates out-of-sample performance.

• Feature Availability Audit
  • The guarding discipline is to define the enemy explicitly as target leakage

    This sourceDefines target leakage as illegitimate target information and combines formal review with methods for detecting leakage in observed data.

• Normalization & Scaling Pipeline
  • Its signature failure is preprocessing leakage — fitting the scaler on the full dataset, so the mean or range used to transform training rows already reflects the test data, quietly inflating validation scores.

Verification

This reference passed the adversarial substantiation pipeline: it was checked to exist and to support the claim it is attached to. See how references were verified.

Registry ID ref:a09e5a6a7c55 · see in the full table