Bias in Error Estimation when Using Cross-validation for Model Selection¶
Varma, S., & Simon, R. (2006). Bias in Error Estimation when Using Cross-validation for Model Selection. BMC Bioinformatics, 7(91).
Cited by¶
1 citation across 1 artifact.
Each citation links to the sentence it supports in the citing article.
Primes¶
- Holdout Set
- The single-use accounting is the formal corrective — each peek spends evaluative power, and nested cross-validation (an outer fold protecting against inner-fold leakage) is the bookkeeping that tracks the erosion.
This sourceDemonstrates the optimistic bias of using one cross-validation loop for both selection and evaluation, and that nested cross-validation (an outer fold) gives an unbiased estimate — the bookkeeping of holdout erosion.
Supported in partVerified against the work's full text
Varma and Simon show reusing CV for tuning and evaluation biases error estimates and that nested CV (inner tuning, outer estimate) corrects it; 'peek' accounting is not their language.
“Thus there are two CV loops; the inner loop is part of the wrapper algorithm and the outer loop computes an estimate of the true error.”
- The single-use accounting is the formal corrective — each peek spends evaluative power, and nested cross-validation (an outer fold protecting against inner-fold leakage) is the bookkeeping that tracks the erosion.
Verification¶
Does it exist? Confirmed. This work's DOI resolves to a registered record, which fixes its identity. That is all it fixes.
Does it back the claim? Read against the text for 1 of 1 citation: 1 supported in part. Each verdict is shown under its citation below, with what in the work backs the sentence.
Support is checked per citation rather than per work — the same source can be cited soundly in one article and wrongly in another. Per-citation recording began recently, so a citation with no recorded check is a gap in the record rather than evidence it went unchecked.
See how references were verified.
Registry ID ref:077354059645 · see in the full table