'Everyone Wants to Do the Model Work, Not the Data Work'¶
Sambasivan, N., Kapania, S., Highfill, H., Akrong, D., Paritosh, P., & Aroyo, L. M. (2021). 'Everyone Wants to Do the Model Work, Not the Data Work': Data Cascades in High-Stakes AI. Proceedings of CHI 2021.
Cited by¶
4 citations across 4 artifacts.
Each citation links to the sentence it supports in the citing article.
Primes¶
- Elicitation Channel Contribution
- Data labelling: the rubric is the channel — the same image under different rubrics yields systematically different labels, and a model trained on the labels learns channel plus signal jointly.
This sourceShows that annotation rubrics systematically shape labels, so a model trained on them learns channel-plus-signal jointly.
- Data labelling: the rubric is the channel — the same image under different rubrics yields systematically different labels, and a model trained on the labels learns channel plus signal jointly.
- Evidence-Fidelity Decay
- in code review written hours after reading a diff, in medical chart documentation written at end of shift, in diplomatic cables and meeting minutes drafted hours after a meeting, and in machine-learning data labelling under time pressure, where labels produced after batch fatigue blend observation of the artefact with reconstruction from the labeller's developing schema.
This sourceDocuments how under-valued, time-pressured data/labeling work produces compounding downstream 'data cascades,' so label sets pass surface checks while encoding annotators' developing schemas rather than the artifact.
- in code review written hours after reading a diff, in medical chart documentation written at end of shift, in diplomatic cables and meeting minutes drafted hours after a meeting, and in machine-learning data labelling under time pressure, where labels produced after batch fatigue blend observation of the artefact with reconstruction from the labeller's developing schema.
Domain-specific¶
- Annotation Drift
- The drift accumulates through onboarding of annotators with different priors, case-law-style internal precedents that accumulate outside the rubric document, ambient context changes that make previously borderline cases seem obvious, and feedback from downstream model errors that reshapes what the team counts as a hard case.
Supported in partVerified against the publisher's abstract
“We define, identify, and present empirical evidence on Data Cascades—compounding events causing negative, downstream effects from data issues—triggered by conventional AI/ML practices that undervalue data quality.”
- The drift accumulates through onboarding of annotators with different priors, case-law-style internal precedents that accumulate outside the rubric document, ambient context changes that make previously borderline cases seem obvious, and feedback from downstream model errors that reshapes what the team counts as a hard case.
- Ground-Truth Drift
- Catching it requires institutional inspection of the labeling process, not statistical monitoring of model outputs
This sourceInterview study across 53 AI practitioners finding that data-quality defects in high-stakes AI originate in undervalued upstream data practices and produce 'invisible, delayed' downstream cascades.
Supported in partVerified against the publisher's abstract
“We define, identify, and present empirical evidence on Data Cascades—compounding events causing negative, downstream effects from data issues—triggered by conventional AI/ML practices that undervalue data quality. Data cascades are pervasive (92% prevalence), invisible, delayed, but often avoidable.”
- Catching it requires institutional inspection of the labeling process, not statistical monitoring of model outputs
Verification¶
Does it exist? Confirmed. This work's DOI resolves to a registered record, which fixes its identity. That is all it fixes.
Does it back the claim? Read against the text for 2 of 4 citations: 2 supported in part. Each verdict is shown under its citation below, with what in the work backs the sentence.
Support is checked per citation rather than per work — the same source can be cited soundly in one article and wrongly in another. Per-citation recording began recently, so a citation with no recorded check is a gap in the record rather than evidence it went unchecked.
See how references were verified.
Registry ID ref:f9b926cbf4ce · see in the full table