Get Another Label? Improving Data Quality and Data Mining Using Multiple, Noisy Labelers

Sheng, V. S., Provost, F., & Ipeirotis, P. G. (2008). Get Another Label? Improving Data Quality and Data Mining Using Multiple, Noisy Labelers. Proceedings of KDD 2008, 614-622.

Type

Conference paper

Intellectual base

Primary research

Year

2008

Pages

614-622

DOI

10.1145/1401890.1401965

DOI registrant

Association for Computing Machinery

Link

https://doi.org/10.1145/1401890.1401965

Cited from

statistics_experimental_design

Cited by

2 citations across 2 artifacts.

Each citation links to the sentence it supports in the citing article.

Primes

• Instrument Interpretive Drift
  • In machine-learning annotation, human labellers applying a constant guide produce label distributions that drift over months as the cohort turns over, internal precedent accumulates, and downstream feedback reshapes habits — while inter-annotator agreement at any one slice stays high.

    This sourceEstablishes that human-labeller annotation quality and agreement are not stable and that label quality shifts with labeller composition and conditions.

Domain-specific

• Proactive learning
  • Work on multiple noisy labelers, including Sheng, Provost, and Ipeirotis, demonstrates that label-source quality and repeated labeling can materially affect data quality.

Verification

Does it exist? Confirmed. This work's DOI resolves to a registered record, which fixes its identity. That is all it fixes.

Does it back the claim? Not recorded. Neither this nor any other of the 2 citations of this work carries a recorded support check.

Support is checked per citation rather than per work — the same source can be cited soundly in one article and wrongly in another. Per-citation recording began recently, so a citation with no recorded check is a gap in the record rather than evidence it went unchecked.

See how references were verified.

Registry ID ref:714272661c3d · see in the full table