On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation.¶
Cawley, G. C., & Talbot, N. L. C. (2010). On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation. Journal of Machine Learning Research, 2079-2107.
Cited by¶
2 citations across 2 artifacts.
Each citation links to the sentence it supports in the citing article.
Mechanisms¶
- Model Tuning Loop
- Its failure mode is overfitting — improving the validation score while the model gets worse at the real task, because the loop has optimized the measurement instead of the target.
This sourceShows that repeatedly optimizing a model-selection criterion can overfit that criterion and substantially degrade generalization performance.
- Its failure mode is overfitting — improving the validation score while the model gets worse at the real task, because the loop has optimized the measurement instead of the target.
- Nested Cross-Validation
- Reusing the same folds for both means the winning configuration has already been fitted to those folds, and the reported score is optimistically biased.
This sourceShows that optimizing and evaluating a model on the same resampling results can overfit the selection criterion and produce optimistically biased performance estimates.
- Reusing the same folds for both means the winning configuration has already been fitted to those folds, and the reported score is optimistically biased.
Verification¶
Does it exist? Not checked yet. This entry carries no identifier to resolve. It was extracted from the citation as written in the article, normalized, and deduplicated against the rest of the registry.
Does it back the claim? Not recorded. Neither this nor any other of the 2 citations of this work carries a recorded support check.
Support is checked per citation rather than per work — the same source can be cited soundly in one article and wrongly in another. Per-citation recording began recently, so a citation with no recorded check is a gap in the record rather than evidence it went unchecked.
See how references were verified.
Registry ID ref:79a9c1d9c65b · see in the full table