Measuring the Reliability of Hate Speech Annotations

Ross, B. (2016). Measuring the Reliability of Hate Speech Annotations: The Case of the European Refugee Crisis.

Type

Preprint

Intellectual base

Grey literature

Year

2016

DOI

10.48550/arxiv.1701.08118

Link

https://doi.org/10.48550/arXiv.1701.08118

Cited by

1 citation across 1 artifact.

Each citation links to the sentence it supports in the citing article.

Domain-specific

• Label Ambiguity
  • Labels like "hate speech," "harassment," and "misinformation" have genuinely fuzzy boundaries, and datasets built for these tasks routinely report low inter-annotator agreement on the contested cases

    This sourceA German hate-speech corpus study finding very low annotation reliability (Krippendorff's α of .18–.38) and concluding that hate speech is a vague concept needing sharper definitions.

    Supported in partVerified against the work's full text

    “Overall, agreement was very low, ranging from α = .18 to .29.”

    Read by an automated reader; not a human review. How support was checked

Verification

Does it exist? Not checked yet. This work's DOI is recorded above but has not been resolved against an external catalogue, so nothing here confirms the work exists.

Does it back the claim? Read against the text for 1 of 1 citation: 1 supported in part. Each verdict is shown under its citation below, with what in the work backs the sentence.

Support is checked per citation rather than per work — the same source can be cited soundly in one article and wrongly in another. Per-citation recording began recently, so a citation with no recorded check is a gap in the record rather than evidence it went unchecked.

See how references were verified.

Registry ID ref:9f1073ab332b · see in the full table