Underspecification Presents Challenges for Credibility in Modern Machine Learning.¶
D'Amour, A. (2022). Underspecification Presents Challenges for Credibility in Modern Machine Learning. Journal of Machine Learning Research, 23(226), 1-61.
Cited by¶
2 citations across 2 artifacts.
Each citation links to the sentence it supports in the citing article.
Primes¶
- Underspecification
- In machine learning, multiple models with identical validation performance encode different decision surfaces; on a stress test — a subgroup, a shifted input, an adversarial probe — they diverge sharply, and the choice among them was governed by initialization or training noise.
This sourceShows that models with identical held-out performance encode different decision surfaces and diverge sharply under stress tests, the closure being seed and training noise.
- In machine learning, multiple models with identical validation performance encode different decision surfaces; on a stress test — a subgroup, a shifted input, an adversarial probe — they diverge sharply, and the choice among them was governed by initialization or training noise.
Domain-specific¶
- Distributional Blind Spot
- The apparent reliability of the model, established by strong validation metrics on the held-out development cohort, actively conceals the failure in the deployment population
This sourceThe finding that predictors with equivalently strong held-out performance are often treated as equivalent yet can behave very differently in deployment domains.
SupportedVerified against a saved copy of the source
“An ML pipeline is underspecified when it can return many predictors with equivalently strong held-out performance in the training domain.”
- The apparent reliability of the model, established by strong validation metrics on the held-out development cohort, actively conceals the failure in the deployment population
Verification¶
Does it exist? Not checked yet. This entry carries no identifier to resolve. It was extracted from the citation as written in the article, normalized, and deduplicated against the rest of the registry.
Does it back the claim? Read against the text for 1 of 2 citations: 1 supported. Each verdict is shown under its citation below, with what in the work backs the sentence.
Support is checked per citation rather than per work — the same source can be cited soundly in one article and wrongly in another. Per-citation recording began recently, so a citation with no recorded check is a gap in the record rather than evidence it went unchecked.
See how references were verified.
Registry ID ref:81a4bbc656a7 · see in the full table