Get Another Label? Improving Data Quality and Data Mining Using Multiple, Noisy Labelers

Sheng, V. S., Provost, F., & Ipeirotis, P. G. (2008). Get Another Label? Improving Data Quality and Data Mining Using Multiple, Noisy Labelers. Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 614-622.

Type

Conference paper

Intellectual base

Primary research

Year

2008

DOI

10.1145/1401890.1401965

Link

https://doi.org/10.1145/1401890.1401965

Cited from

statistics_experimental_design

Cited by

2 citations across 2 artifacts.

Each citation links to the sentence it supports in the citing article.

Primes

• Instrument Interpretive Drift
  • In machine-learning annotation, human labellers applying a constant guide produce label distributions that drift over months as the cohort turns over, internal precedent accumulates, and downstream feedback reshapes habits — while inter-annotator agreement at any one slice stays high.

    This sourceEstablishes that human-labeller annotation quality and agreement are not stable and that label quality shifts with labeller composition and conditions.

Domain-specific

• Proactive learning
  • … survey locates this problem within active learning, where selective label acquisition aims to reduce labeling effort while preserving model quality. Work on multiple noisy labelers, including Sheng, Provost, and Ipeirotis, demonstrates that label-source quality and repeated labeling can materially affect data quality.

Verification

This reference passed the adversarial substantiation pipeline: it was checked to exist and to support the claim it is attached to. See how references were verified.

Registry ID ref:714272661c3d · see in the full table