Inter-Coder Agreement for Computational Linguistics¶
Artstein, & Poesio. (2008). Inter-Coder Agreement for Computational Linguistics. Computational Linguistics.
Cited by¶
2 citations across 2 artifacts.
Each citation links to the sentence it supports in the citing article.
Domain-specific¶
- Inter-Annotator Agreement
- The canonical statistics are Cohen's kappa (two annotators), Fleiss's kappa (multiple annotators over the same items), and Krippendorff's alpha (handles missing data and multiple scale types)
This sourceArtstein and Poesio's survey, which treats Cohen's kappa, the multi-coder generalisations and Krippendorff's alpha, the last accommodating multiple coders, graded disagreement, missing values and several scale types.
Supported in partVerified against the work's full text
“A generalization of Scott’s π is proposed in Fleiss (1971), and a generalization of Cohen’s κ is given in Davies and Fleiss (1982).”
- The canonical statistics are Cohen's kappa (two annotators), Fleiss's kappa (multiple annotators over the same items), and Krippendorff's alpha (handles missing data and multiple scale types)
- Label Ambiguity
- The interventions — multi-annotator adjudication protocols, soft labels encoding the agreement distribution, conformal abstention in the disagreement zone, Krippendorff's alpha and Cohen's kappa as explicit reporting requirements
This sourceThe survey's recommendation that annotation efforts should perform and report rigorous reliability testing, using Cohen's κ and Krippendorff's α.
Supported in partVerified against the work's full text
“Our first recommendation is that annotation efforts should perform and report rigorous reliability testing.”
- The interventions — multi-annotator adjudication protocols, soft labels encoding the agreement distribution, conformal abstention in the disagreement zone, Krippendorff's alpha and Cohen's kappa as explicit reporting requirements
Verification¶
Does it exist? Confirmed. This work's DOI resolves to a registered record, which fixes its identity. That is all it fixes.
Does it back the claim? Read against the text for 2 of 2 citations: 2 supported in part. Each verdict is shown under its citation below, with what in the work backs the sentence.
Support is checked per citation rather than per work — the same source can be cited soundly in one article and wrongly in another. Per-citation recording began recently, so a citation with no recorded check is a gap in the record rather than evidence it went unchecked.
See how references were verified.
Registry ID ref:6d3bf374df19 · see in the full table