Cohen's Kappa¶

A two-labeler nominal agreement coefficient that normalizes observed matching above the agreement expected from their separate category frequencies.

Version: v1 · 2026-10-17 · History
Domain-specific #: 14089
Domain group: Formal Sciences
Origin domain: Experimental Design & Statistics
Subdomain: Categorical Agreement → Experimental Design & Statistics
Aliases: Cohen's kappa coefficient, Unweighted Cohen's kappa

Core Idea¶

Cohen's kappa is a coefficient for two complete nominal-category labelings of the same items. It compares the proportion of items on which the two assignments match with the agreement expected from the product of the labelers' separate observed category frequencies, then divides that excess by the remaining span above the expected level. The calculation has a defined value only when expected agreement is below one. The original named coefficient was introduced by Cohen in 1960; the formula used here is checked in an independently inspectable research paper, not attributed to uninspected pages of Cohen's original article.^[1]^[2]

The result answers a narrow question: How much of the available agreement above this marginal-product reference appears in the paired table? It is neither a truth test for the categories nor a certificate that the labelers are accurate. The same computation can summarize two radiologists' category judgments or two linguists' annotations, even though those practices differ. A category scheme, a pair of corresponding assignments, and the two separate margins must remain visible when interpreting the number.^[2]^[3]^[4]

Structural Signature¶

Sig role-phrases:

R1 — paired items and aligned nominal categories: each assessed unit must have corresponding assignments under category meanings that can be matched; otherwise the diagonal of an agreement table has no interpretation.
R2 — two complete labelings: the first labeling supplies rows and the second columns. Three readers can yield several two-reader comparisons, but their ratings do not automatically make one three-reader Cohen coefficient.^[3]
R3 — observed diagonal share, p_o: matched category assignments divided by the total number of paired items.
R4 — separate margins and reference share, p_e: for each category, multiply the first labeler's observed fraction by the second's, then add those products. This is the expected-match term under a reference model of independent draws at those empirical rates, not an observation of independently repeated labeling.^[2]
R5 — excess-agreement normalization: divide p_o − p_e by 1 − p_e when p_e < 1. Without this denominator the result is not the named coefficient; when p_e = 1, the displayed ratio is undefined.

These are computational roles. They do not require the labelers to be human, to have been trained together, or to make their actual judgments independently in a causal sense. The independence assumption belongs to the reference calculation for p_e; whether the observed judgments were collected under an appropriate protocol is a separate study-design question.^[2]

What It Is Not¶

Not raw percent agreement. Percent agreement is p_o alone. Kappa adds a specified comparison with agreement implied by the two empirical margins. Two tables with the same raw match rate can therefore have different kappa values if their category-use margins differ.^[2]

Not a pooled-marginal or weighted variant. In the two-category comparison by Powers, Scott-style common-distribution expectation differs from Cohen's product of two separate margins. Cohen also published a separately named weighted, partial-credit variant. These are related method identities rather than the unweighted coefficient defined here; this entry does not assert their full formulas.^[5]^[6]

Not accuracy, validity, or a universal pass threshold. Two labelers can agree on a mistaken category assignment. A high kappa therefore cannot by itself establish diagnostic truth, category validity, or permission to release a dataset. Conversely, a low value alone does not identify whether the source is category prevalence, different marginal habits, a difficult item set, or a defective coding guide.^[2]^[3]

Not one multi-rater statistic. With three radiologists, calculate and name each two-reader pairing if using this coefficient; do not collapse all ratings into an unqualified single Cohen value. The MRI study reports precisely three initial pairwise coefficients.^[3]

Scope of Application¶

The literal habitat is paired categorical assessment: the same units receive two comparable nominal assignments, making a square category-by-category table possible. The statistical relation can be used within clinical imaging, linguistic annotation, content coding, or classifier/reference comparisons. Its computational reach does not depend on the institution or physical substrate of the assessed items.^[2]^[3]^[4]

The published imaging example is a 20-exam Modic-grading study set in which three radiologists were compared pairwise. The paper describes three binary Modic-presence labels per vertebral-body bisection, reports the pairwise kappa values, and does not publish the underlying pair tables or the exact unit used for each coefficient. It supports a reported application, not a reconstructed worked calculation.^[3]

The linguistic example gives a different, fully tabulated use: pairs of annotators classify transitivity annotations, and one reported two-by-two table permits the entire calculation to be reproduced. Five annotator pairs in that one paper are still one linguistic setting, not five unlike domains.^[4]

Clarity¶

The distinction between p_o and p_e prevents the word “agreement” from hiding two different quantities. p_o describes what the paired assignments did. p_e describes what a specific independent-draw calculation would expect if each labeler retained their observed category proportions. Calling p_e “chance agreement” is conventional shorthand; it does not mean that anyone observed the same labelers guessing at random.^[2]

The separate margins matter. If one labeler uses a category more often than the other, their marginal vectors differ and the expected-match products reflect that asymmetry. Reporting only kappa suppresses the diagonal match rate and the category mix that generated it; reporting only percent agreement suppresses the reference. When p_e = 1, the formula divides by zero, so a perfectly uniform one-category table should be described as degenerate rather than given an ordinary kappa of one.^[2]

Manages Complexity¶

A paired categorical dataset can contain many item-level judgments and many cells of a contingency table. Kappa reduces one aspect of that table to three connected quantities: the diagonal share, the product-of-margins reference, and the normalized excess. That reduction makes a pairwise comparison portable across coding tasks without requiring a new agreement concept for each category name.^[2]

The compression is deliberately incomplete. Different tables can yield the same coefficient while differing in class balance, disagreement pattern, and the consequences of particular errors. A useful report retains the table or at least p_o and the margins when possible. The coefficient does not replace an error analysis, a validity study, or an assessment of whether the categories were applied to genuinely corresponding units.

Abstract Reasoning¶

Let n_ij count units that labeler 1 assigns category i and labeler 2 assigns category j, and let N be the total number of paired units. The row total n_i+ and column total n_+i produce the two empirical category proportions. For the unweighted nominal coefficient,

\[ p_o=\frac{\sum_i n_{ii}}{N},\qquad p_e=\sum_i\frac{n_{i+}}{N}\frac{n_{+i}}{N},\qquad \kappa=\frac{p_o-p_e}{1-p_e}\quad(p_e<1). \]

This is the calculation stated by Bruce and Wiebe. At fixed p_e < 1, observed agreement equal to the reference gives zero, observed agreement of one gives one, and observed agreement below the reference gives a negative result. These are algebraic relationships, not qualitative labels such as “adequate” or “excellent.” If the denominator vanishes, the displayed coefficient is undefined rather than an ordinary limiting verdict.^[2]

The inference the coefficient permits is conditional: given this paired table, this common category scheme, and this marginal-product reference, it quantifies excess matching. It does not establish why the match occurred or whether either labeler found a true category.

Knowledge Transfer¶

The transfer from MRI grading to linguistic annotation preserves the same computational roles: corresponding units, two assignments, diagonal matching, two empirical category distributions, and a normalized excess. The domain changes the meaning and reliability of the labels, not the algebra of the coefficient.^[3]^[4]

What does not transfer automatically is a numerical benchmark or interpretation rule. One corpus can be strongly imbalanced while another is more evenly labeled; two coefficient values then cannot be ranked as if they alone summarize all aspects of labeling quality. A clinical pairwise score also cannot establish diagnostic accuracy, just as an annotation pairwise score cannot by itself approve a coding scheme for release. Those decisions require domain-specific evidence in addition to the coefficient.

Examples¶

Canonical: a fully tabulated linguistic annotation pair¶

Dong, Liu and Fang report transitivity annotation by five pairs of annotators. For Pair 5, their Table 5 gives cells 67, 1, 4, 13 in a two-by-two normal/abnormal table, totaling 85 items. The matched diagonal contains 67 + 13 = 80, so p_o = 80/85. Annotator A's row totals are 68 and 17; B's column totals are 71 and 14. Therefore p_e = (68×71 + 17×14)/85² = 5066/7225, and the unweighted formula gives κ = 102/127 ≈ 0.80315. The authors report the rounded value 0.80 in Table 4. The arithmetic is our reconstruction from their original table and the separately inspected formula.^[4]^[2]

Mapped back: R1 is the 85 common annotation units and their two aligned labels; R2 is the A/B pair; R3 is the 80 matched cells; R4 is the two distinct margin vectors and their 5066/7225 product sum; R5 turns their difference into the reported rounded coefficient. The authors' qualitative descriptor for 0.80 is local to their study and is not made a universal threshold here.

Applied: pairwise MRI grading¶

Gao and colleagues assessed Modic grading on a separately curated set of 20 lumbar MRI exams. Three radiologists graded independently, then two readers participated in a later AI-assisted phase. Before assistance, the paper reports kappa values of 0.63 for readers ½, 0.52 for ⅓, and 0.45 for ⅔. The later assisted values and a different model-versus-reference comparison are separate analyses.^[3]

Mapped back: R1 is corresponding graded material within the shared study set, whose exact per-coefficient analysis unit is not published; R2 is each named two-reader comparison; R3 and R4 must underlie the reported coefficients but their raw cells and marginals are unavailable; R5 is each reported initial pairwise value. This example demonstrates an unlike application. It does not authorize a fabricated p_o, p_e, or contingency table.^[3]

Structural Tensions¶

T1: Visible matching versus marginal-adjusted matching. A simple observed match rate is easy to communicate, while kappa asks what remains above the reference implied by the two label distributions. Favoring raw matching alone can make a skewed table look more informative than it is; favoring one corrected number alone can conceal the table and marginal mix that explain it. Neither summary should be maximized or interpreted in isolation as a quality verdict. Diagnostic: For this paired table, do the margins make raw matching misleading enough that both p_o and kappa should be reported?^[2]

T2: Agreement summary versus validity decision. A compact pairwise coefficient helps compare labeling behavior, but validating the category meanings or releasing a coding procedure requires evidence outside its formula. Treating agreement as the whole acceptance test risks certifying shared error; ignoring it because external validation exists misses reproducibility problems. Diagnostic: Is this number being used only to report pairwise convergence, or is it being asked to decide truth or release without a separate standard?^[2]^[3]

Structural–Framed Character¶

Cohen's kappa has a formal structural computation inside a framed measurement practice. Once a paired table and nominal category correspondence are fixed, its diagonal and marginal arithmetic are exact. The coefficient itself does not favor a clinical diagnosis, an annotation label, or a social judgment; its evaluative weight enters when someone treats a value as acceptable or unacceptable.^[2]

Human practice supplies the units, categories, observations, labeling protocol, and purpose of interpretation. Cohen's named formula arose within statistical methodology rather than from a natural object's intrinsic label, but its use travels literally among different coding habitats. Applying it to MRI assessments and language annotations is recognition of the same computation, not a metaphor. Calling a merely similar notion of “agreement” kappa without paired categories and the separate-marginal calculation would instead import its vocabulary without its mechanism. Its character: a mathematically fixed transformation whose instances and evaluative use depend on a human-specified category scheme and measurement setting.^[1]^[3]^[4]

Structural Core vs. Domain Accent¶

The core is the paired-table transformation: diagonal matches are evaluated against a product-of-separate-margins reference and normalized by the span above that reference. Category names, annotator professions, software, and study objectives vary while that relation remains fixed.^[2]

The generic input-to-output skeleton has portable reach in the existing Function (Mapping) Prime: a specified paired table with p_e < 1 deterministically maps to one real coefficient. That pattern is the approved strict Function (Mapping) parent of the type-level coefficient: the separate-marginal reference and normalization are its stable statistical differentia. The named entry remains within categorical statistical measurement: remove paired nominal assignments and their marginal-product reference, and the coefficient disappears. Its reuse in medicine and linguistics does not create a substrate-general Prime about all forms of agreement. The broader live Inter-Annotator Agreement entry addresses a coding-scheme reliability workflow with several possible statistics; this entry isolates one coefficient that can be computed without undertaking that whole workflow. Neither identity is reducible to a duplicated formula paragraph in the other.

This entry is a kind of Function (Mapping).

The coefficient is a strict instance of Function (Mapping): its admissible domain is paired nominal tables with N > 0 and p_e < 1, its codomain is real coefficients, and its rule is single-valued and independent of ambient interpretation. Its stable differentia is the observed diagonal share, the two separate empirical margins, and normalized excess matching. Function (Mapping) also occurs without any of those statistical preconditions. This is the sole approved typed parent. Classification remains a related prerequisite at the level of supplied labels, but the coefficient does not itself perform category assignment; its complete live signature does not justify an additional strict edge.

The live Inter-Annotator Agreement entry names this coefficient as one tool, yet its whole workflow includes scheme training, reliability assessment, and decisions about a coding process. Those practices may use Cohen's kappa, Fleiss-like or other statistics. This entry's two-labeling arithmetic is narrower and can also be applied when the paired outputs are a classifier and a reference rather than two human annotators. The distinction is coefficient versus the larger assessment process.

Relationships to Other Abstractions¶

Current abstraction Cohen's Kappa Domain-specific

Parents (1) — more general patterns this builds on

Cohen's Kappa is a kind of Function (Mapping) Prime

Cohen's kappa is a single-valued mapping from admissible two-labeler nominal tables to a real coefficient; the separate-marginal reference and normalization are its stable differentia.

Hierarchy path (1) — routes to 1 parentless root

Cohen's Kappa → Function (Mapping)

Neighborhood in Abstraction Space¶

Cohen's Kappa sits in a sparse region of the domain-specific corpus (69^th percentile for distinctiveness): few abstractions share its structure, so a faithful description tends to retrieve it precisely.

Family — Unclustered & Miscellaneous (2721 abstractions)

Nearest neighbors

Computed from structural-signature embeddings · 2026-10-17

Not to Be Confused With¶

A raw agreement percentage: omits the separate-marginal reference.^[2]
Scott-style pooled-marginal agreement: in the inspected two-class comparison, the common-distribution reference differs from Cohen's two separate margins.^[5]
Weighted kappa: Cohen's separately named partial-credit variant; its detailed formula is outside this unweighted entry.^[6]
Fleiss-like multi-rater agreement: a distinct extension or alternative when the target is one coefficient across several raters, not three pairwise Cohen values.^[2]^[3]
Diagnostic accuracy or label validity: agreement between two outputs can coexist with common error.
An ordinary score when p_e = 1: the displayed ratio has a zero denominator.
The live K Statistic or Kappa Effect entries: their names use similar letters but their actual identities are different.

References¶

[1] Jacob Cohen (1960). “A Coefficient of Agreement for Nominal Scales”. Educational and Psychological Measurement 20(1), 37–46. Publisher metadata and abstract inspected for bibliographic origin only; original full pages were not available for claim verification. registry ↩a ↩b

[2] Rebecca Bruce and Janyce Wiebe (1998). “Word-Sense Distinguishability and Inter-Coder Agreement”. Proceedings of the Third Conference on Empirical Methods for Natural Language Processing, pp. 53–60, especially “The Basics” and Equation (1), printed pp. 54–55. Full original research PDF inspected for the paired nominal table, separate-marginal reference, and coefficient formula. registry ↩a ↩b ↩c ↩d ↩e ↩f ↩g ↩h ↩i ↩j ↩k ↩l ↩m ↩n ↩o ↩p ↩q ↩r

[3] Kenneth T. Gao and colleagues (2022). “Automatic detection and voxel-wise mapping of lumbar spine Modic changes with deep learning”. JOR Spine 5(2), e1204. Methods §2.3.1, Results §3.4 and Table 3. Full original research article inspected; pair tables and exact coefficient analysis units are not reported. registry ↩a ↩b ↩c ↩d ↩e ↩f ↩g ↩h ↩i ↩j ↩k ↩l

[4] Min Dong, Xiaoyan Liu and Alex Chengyu Fang (2022). “Testing the Annotation Consistency of Hallidayan Transitivity Processes: A Multi-variable Structural Approach”. Proceedings of ISA-18 Workshop at LREC 2022, pp. 53–60. Sections 3–4, Tables 4–5, especially printed p. 57 / PDF index 4. Full original research PDF inspected; the Pair 5 calculation above is reconstructed from its original table. registry ↩a ↩b ↩c ↩d ↩e ↩f

[5] David Martin Ward Powers (2012). “The Problem with Kappa”. Proceedings of the 13^th Conference of the European Chapter of the Association for Computational Linguistics, pp. 345–355, §1.1, especially printed p. 347 / PDF index 2. Full original method-comparison PDF inspected for the bounded two-class Cohen versus Scott-style reference contrast. registry ↩a ↩b

[6] Jacob Cohen (1968). “Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit”. Psychological Bulletin 70(4), 213–220, DOI 10.1037/h0026256. PubMed original-work metadata/title inspected only; no claim of full-paper formula review. registry ↩a ↩b