The Reusable Holdout¶
Dwork, C., Feldman, V., Hardt, M., Pitassi, T., Reingold, O., & Roth, A. (2015). The Reusable Holdout: Preserving Validity in Adaptive Data Analysis. Science, 349(6248), 636-638.
Cited by¶
3 citations across 3 artifacts.
Each citation links to the sentence it supports in the citing article.
Mechanisms¶
- Fresh Holdout Retest
- The discipline comes from the reusable-holdout literature
This sourceIntroduces a reusable holdout that preserves validity under repeated adaptive use of holdout data.
- The discipline comes from the reusable-holdout literature
- Held-Out Benchmark Dataset
- This is deliberate: competitors who probe the public leaderboard hundreds of times will drift toward overfitting it, and the untouched private split is what catches them.
This sourceShows that repeated adaptive feedback from a holdout or leaderboard can cause overfitting to that benchmark and that evaluation on fresh, protected data exposes the resulting failure to generalize.
- This is deliberate: competitors who probe the public leaderboard hundreds of times will drift toward overfitting it, and the untouched private split is what catches them.
- Holdout Validation
Verification¶
Does it exist? Confirmed. This work's DOI resolves to a registered record, which fixes its identity. That is all it fixes.
Does it back the claim? Not recorded. Neither this nor any other of the 3 citations of this work carries a recorded support check.
Was it audited? Yes. A second, independent pass read the citation against the article text and recorded a verdict.
Support is checked per citation rather than per work — the same source can be cited soundly in one article and wrongly in another. Per-citation recording began recently, so a citation with no recorded check is a gap in the record rather than evidence it went unchecked.
See how references were verified.
Registry ID ref:35262a4a7761 · see in the full table