Cohen's Kappa¶
A two-labeler nominal agreement coefficient that normalizes observed matching above the agreement expected from their separate category frequencies.
Core Idea¶
Cohen's kappa summarizes how much two sets of category labels match beyond the agreement expected from each labeler's own category frequencies. Both labelers must assess corresponding items using comparable nominal categories. Kappa starts with the fraction of matching labels, subtracts the expected-match fraction calculated from the two separate frequency distributions, and divides by the fraction of agreement still possible above that expectation. It is defined by this formula only when the expected fraction is below one.[^ref-a9992e2d26bf]
The number describes agreement under that specific reference calculation. It does not say that either labeler chose the correct categories. Cohen introduced the named nominal-scale coefficient in 1960; the formula used here is supported by a separately inspected original research paper.[ref-8fbbed32ed6c][ref-a9992e2d26bf]
Scope of Application¶
The calculation can be used when two people label the same text items or two radiologists grade corresponding medical-imaging material. It can also compare categorical outputs from a model and a reference, provided the units and category meanings line up. These are uses of one calculation, not evidence that it validates the labels themselves.[ref-a9992e2d26bf][ref-64420911e8c3][^ref-efb462cbb246]
In one lumbar MRI study, three radiologists were assessed in three separate pairs before a later AI-assisted phase. Their initial pairwise kappa values were 0.63, 0.52, and 0.45. The paper reports these values but not the raw pair tables or the exact unit used for each coefficient, so its expected-agreement values cannot be reconstructed from the publication.[^ref-64420911e8c3]
Clarity¶
Keep two quantities separate. Observed agreement, p_o, is the fraction of paired assignments that match. Expected agreement, p_e, is the sum of products of the two labelers' category proportions under an independent-draw reference calculation. Calling p_e “chance agreement” does not mean the observed labelers were seen guessing randomly.[^ref-a9992e2d26bf]
Raw percent agreement reports only p_o. A pooled-marginal method changes the reference, and a separately named weighted kappa allows partial credit; neither is the unweighted calculation here. A value of kappa is also not an accuracy or validity verdict. When p_e = 1, the displayed formula has a zero denominator and should not be presented as an ordinary score.[ref-a9992e2d26bf][ref-f41cffae0165][^ref-7ff258c87959]
Manages Complexity¶
Many paired labeling decisions can be arranged in a category-by-category table. Kappa reduces that table to the matching diagonal, the two category-frequency margins, and one normalized result. This helps an analyst compare two labelings without reviewing every item individually. The reduction does not preserve which categories were confused, so the table and margins are still useful when explaining a result.[^ref-a9992e2d26bf]
Abstract Reasoning¶
For a table with N paired items, count diagonal matches to obtain p_o. For each category, multiply labeler A's fraction by labeler B's fraction, and add the products to obtain p_e. Then calculate κ = (p_o − p_e)/(1 − p_e) if p_e < 1.[^ref-a9992e2d26bf]
The linguistic study by Dong, Liu and Fang publishes an auditable example. One pair's two-by-two cells are 67, 1, 4, 13 across 85 items. The matching share is 80/85; the separate margins are 68/17 and 71/14; expected agreement is 5066/7225. These yield κ = 102/127 ≈ 0.80315, matching the paper's rounded 0.80. This computation comes from its published table, unlike the MRI scores whose underlying tables are not supplied.[ref-efb462cbb246][ref-a9992e2d26bf]
Knowledge Transfer¶
The MRI and linguistic examples change the subject matter but keep the same roles: corresponding units, two comparable labelings, matched cells, separate category frequencies, and normalization. Their category meanings, collection procedures, and downstream decisions do not transfer with the formula.[ref-64420911e8c3][ref-efb462cbb246]
The live Inter-Annotator Agreement entry addresses a broader coding-scheme reliability practice that can use different statistics. Cohen's kappa is this one coefficient, which can be computed without carrying out that whole workflow. The type-level formula is a strict instance of Function (Mapping): it takes an admissible paired nominal table to one real coefficient. Its separate-marginal reference and normalization make it a more specific statistical rule. That is its sole approved typed parent.
Relationships to Other Abstractions¶
Current abstraction Cohen's Kappa Domain-specific
Parents (1) — more general patterns this builds on
-
Cohen's Kappa is a kind of Function (Mapping) Prime
Cohen's kappa is a single-valued mapping from admissible two-labeler nominal tables to a real coefficient; the separate-marginal reference and normalization are its stable differentia.
Hierarchy path (1) — routes to 1 parentless root
- Cohen's Kappa → Function (Mapping)
Neighborhood in Abstraction Space¶
Cohen's Kappa sits in a sparse region of the domain-specific corpus (69th percentile for distinctiveness): few abstractions share its structure, so a faithful description tends to retrieve it precisely.
Family — Unclustered & Miscellaneous (2721 abstractions)
Nearest neighbors
- Join Count Statistic — 0.86
- Phi Coefficient — 0.85
- Co-citation — 0.84
- Yoked Control Design — 0.83
- Multivariate Glyph — 0.83
Computed from structural-signature embeddings · 2026-10-17
References¶
[^ref-8fbbed32ed6c]: Jacob Cohen (1960). “A Coefficient of Agreement for Nominal Scales”. Educational and Psychological Measurement 20(1), 37–46. Publisher metadata and abstract inspected for bibliographic origin only; original full pages were not available for claim verification. [^ref-a9992e2d26bf]: Rebecca Bruce and Janyce Wiebe (1998). “Word-Sense Distinguishability and Inter-Coder Agreement”. Proceedings of the Third Conference on Empirical Methods for Natural Language Processing, pp. 53–60, especially “The Basics” and Equation (1), printed pp. 54–55. Full original research PDF inspected for the paired nominal table, separate-marginal reference, and coefficient formula. [^ref-64420911e8c3]: Kenneth T. Gao and colleagues (2022). “Automatic detection and voxel-wise mapping of lumbar spine Modic changes with deep learning”. JOR Spine 5(2), e1204. Methods §2.3.1, Results §3.4 and Table 3. Full original research article inspected; pair tables and exact coefficient analysis units are not reported. [^ref-efb462cbb246]: Min Dong, Xiaoyan Liu and Alex Chengyu Fang (2022). “Testing the Annotation Consistency of Hallidayan Transitivity Processes: A Multi-variable Structural Approach”. Proceedings of ISA-18 Workshop at LREC 2022, pp. 53–60. Sections 3–4, Tables 4–5, especially printed p. 57 / PDF index 4. Full original research PDF inspected; the Pair 5 calculation above is reconstructed from its original table. [^ref-f41cffae0165]: David Martin Ward Powers (2012). “The Problem with Kappa”. Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, pp. 345–355, §1.1, especially printed p. 347 / PDF index 2. Full original method-comparison PDF inspected for the bounded two-class Cohen versus Scott-style reference contrast. [^ref-7ff258c87959]: Jacob Cohen (1968). “Weighted kappa: nominal scale agreement with provision for scaled disagreement or partial credit”. Psychological Bulletin 70(4), 213–220, DOI 10.1037/h0026256. PubMed original-work metadata/title inspected only; no claim of full-paper formula review.