AI and the Everything in the Whole Wide World Benchmark.¶
Raji, I. D., Bender, E. M., Paullada, A., Denton, E., & Hanna, A. (2021). AI and the Everything in the Whole Wide World Benchmark. Proceedings of the NeurIPS Datasets and Benchmarks Track.
Cited by¶
2 citations across 2 artifacts.
Each citation links to the sentence it supports in the citing article.
Primes¶
- Construct Validity
- The classical validity battery ports almost unchanged from psychometrics to the evaluation of computational systems: convergent validity becomes "does the benchmark agree with held-out human judgment?", discriminant becomes "is it uncorrelated with confounds like input length?"
This sourceArgues that 'general' AI benchmarks have construct-validity problems, including failure to discriminate the named capability from confounds, and calls for checking benchmark validity rather than assuming it.
- The classical validity battery ports almost unchanged from psychometrics to the evaluation of computational systems: convergent validity becomes "does the benchmark agree with held-out human judgment?", discriminant becomes "is it uncorrelated with confounds like input length?"
- Fallacy Of Misplaced Concreteness
- Statistics and ML — benchmark scores or held-out accuracy mistaken for the underlying capability they were meant to measure.
This sourceArgues benchmark scores are mistaken for the underlying capability they were meant to measure — construct reification in machine learning.
- Statistics and ML — benchmark scores or held-out accuracy mistaken for the underlying capability they were meant to measure.
Verification¶
Does it exist? Not checked yet. This entry carries no identifier to resolve. It was extracted from the citation as written in the article, normalized, and deduplicated against the rest of the registry.
Does it back the claim? Not recorded. Neither this nor any other of the 2 citations of this work carries a recorded support check.
Support is checked per citation rather than per work — the same source can be cited soundly in one article and wrongly in another. Per-citation recording began recently, so a citation with no recorded check is a gap in the record rather than evidence it went unchecked.
See how references were verified.
Links previously used in the corpus¶
Before the registry existed this work was also linked 1 other way.
Registry ID ref:3c1f83d878a0 · see in the full table