Leakage and the Reproducibility Crisis in Machine-Learning-Based Science

Kapoor, S., & Narayanan, A. (2023). Leakage and the Reproducibility Crisis in Machine-Learning-Based Science. Patterns.

Type

Journal article

Intellectual base

Primary research

Year

2023

DOI

10.1016/j.patter.2023.100804

Link

https://doi.org/10.1016/j.patter.2023.100804

Cited from

epistemology

Cited by

1 citation across 1 artifact.

Each citation links to the sentence it supports in the citing article.

Primes

• Past-State Contamination
  • The violation lives in the pairing of a late input with an as-of assertion, and it can be cured from either side: withdraw the input, or withdraw the claim.

    This sourceSurveys leakage across hundreds of papers in many fields, finds it produced by ordinary convenience rather than intent, catalogues preprocessing fitted across the whole dataset and temporal leakage among its types, and shows reported performance collapsing once the leak is closed.

Verification

This reference passed the adversarial substantiation pipeline: it was checked to exist and to support the claim it is attached to. See how references were verified.

Registry ID ref:7fe8b3834630 · see in the full table