Leakage and the Reproducibility Crisis in Machine-Learning-Based Science¶
Kapoor, S., & Narayanan, A. (2023). Leakage and the Reproducibility Crisis in Machine-Learning-Based Science. Patterns.
Cited by¶
2 citations across 2 artifacts.
Each citation links to the sentence it supports in the citing article.
Primes¶
- Past-State Contamination
- The violation lives in the pairing of a late input with an as-of assertion, and it can be cured from either side: withdraw the input, or withdraw the claim.
This sourceSurveys leakage across hundreds of papers in many fields, finds it produced by ordinary convenience rather than intent, catalogues preprocessing fitted across the whole dataset and temporal leakage among its types, and shows reported performance collapsing once the leak is closed.
- The violation lives in the pairing of a late input with an as-of assertion, and it can be cured from either side: withdraw the input, or withdraw the claim.
Domain-specific¶
- Imputation Leakage
- The failure is particularly easy to introduce because imputation is typically thought of as data cleaning rather than modelling, so practitioners who are vigilant about the train/test partition for the main model may not apply the same discipline to preprocessing steps.
This sourceA survey of machine-learning-based science finding data leakage in 294 papers across 17 fields, and listing pre-processing such as imputation on the entire dataset before the split as one of its documented types. A taxonomy of eight types of data leakage across three top-level categories, in which pre-processing such as imputation fitted on the entire dataset before the split is one documented type.
Supported in partVerified against the work's full text
“Using the entire dataset for any pre-processing steps, such as imputation or over/under sampling, results in leakage.”
- The failure is particularly easy to introduce because imputation is typically thought of as data cleaning rather than modelling, so practitioners who are vigilant about the train/test partition for the main model may not apply the same discipline to preprocessing steps.
Verification¶
Does it exist? Confirmed. This work's DOI resolves to a registered record, which fixes its identity. That is all it fixes.
Does it back the claim? Read against the text for 1 of 2 citations: 1 supported in part. Each verdict is shown under its citation below, with what in the work backs the sentence.
Support is checked per citation rather than per work — the same source can be cited soundly in one article and wrongly in another. Per-citation recording began recently, so a citation with no recorded check is a gap in the record rather than evidence it went unchecked.
See how references were verified.
Registry ID ref:7fe8b3834630 · see in the full table