Skip to content

Classification Disagreement Audit

Diagnostic audit — instantiates Prototype-Centered Category Modeling

Measures where classifications diverge — reviewer vs reviewer, human vs model — to expose systematic bias and human-model misalignment.

A Classification Disagreement Audit treats disagreement as data. It takes the same set of cases classified by more than one source — two reviewers, or a human panel and a model — and maps where and how they diverge. What makes it this mechanism rather than a calibration or review sibling is that it is a measurement, not a fix: it does not reconcile the judges or rule on the cases; it quantifies the pattern of disagreement to expose two things a single label hides — where a model has silently parted from human judgement, and where disagreement clusters on an irrelevant attribute in a way that betrays bias. It produces a diagnosis; other mechanisms act on it.

Example

A lender uses a model to pre-screen loan applications into a "high-risk" category, with human underwriters reviewing a share. An audit pulls several thousand recently double-judged applications and lines up the two labels. Two patterns fall out. First, the model and the underwriters agree on the clear cases but diverge sharply in the middle band — evidence the model's prototype of "high-risk" has departed from the humans' exactly where it matters most. Second, when the disagreements are sliced by applicant features, they cluster on postcode in a way that tracks a protected characteristic more than it tracks default history — a bias signal, since postcode is standing in as a proxy.[1] The audit retrains nothing and overrules no single decision; it hands the org a precise map: here the machine is out of step with people, and here the category is leaning on something it should not.

How it works

The distinguishing move is to compute over pairs of judgements on the same cases, not over the cases themselves. Agreement is quantified, and — more usefully — the disagreements are characterised: where in the typicality range they concentrate (usually the marginal band), whether one source runs systematically higher than the other (a calibration gap between human and model), and whether disagreement correlates with any attribute that ought to be irrelevant (a bias flag). The output is a structured description of how the classifications part company, aimed at whoever will fix it.

Tuning parameters

  • Comparison pairing — reviewer-vs-reviewer exposes human inconsistency; human-vs-model exposes model misalignment; each answers a different question, so choose per audit.
  • Slice set — which attributes disagreement is broken down by; the bias findings are only as good as the slices you think to cut.
  • Agreement metric — raw agreement vs. a chance-corrected statistic; the latter avoids crediting agreement that is just both sources following the base rate.
  • Sample focus — auditing a random sample vs. over-sampling the marginal band where disagreement, and bias, concentrate.
  • Significance bar — how large a skew counts as a finding rather than noise; set it to trade false alarms against missed bias.

When it helps, and when it misleads

Its strength is making invisible divergence measurable: a model can pass an accuracy check yet be quietly misaligned with human judgement on exactly the cases that matter, and a category can look neutral while leaning on a biased proxy — the audit surfaces both before they harden into policy. It is the mechanism most likely to catch a prototype that has picked up a stereotype.

It misleads when disagreement is read as fault — a divergence says the two sources differ, not which is right, and treating the human as ground truth can launder human bias into the "correct" answer. Chance agreement can flatter a metric, and a slice you did not cut hides a bias you will never see. The classic misuse is the audit run to exonerate — sliced narrowly and reported selectively to certify a system as fair rather than to test it. The discipline is to pre-register the slices and metrics, treat findings as questions to investigate rather than verdicts, and route what turns up to the mechanisms that actually fix it.

How it implements the components

The audit fills the detection-side components a diagnostic can measure:

  • machine_model_alignment_layer — by pairing model labels against human ones on shared cases, it measures whether the machine's prototype still tracks human judgement, and where it has parted from it.
  • bias_and_stereotype_guardrail — slicing disagreement by attribute is the guardrail's detection mechanism: it surfaces where the category is leaning on something it should not.

It only diagnoses. Reconciling the judges is a Calibration Workshop; ruling on the flagged marginal cases is the Boundary Case Review Panel; and telling genuine change-over-time apart from static disagreement is a Drift Sample Review.

Notes

An audit is point-in-time and diagnostic: it says that and where sources disagree, not why, and never fixes anything itself. Its value is entirely in which downstream mechanism the findings are routed to — an audit whose results nobody acts on has only documented the problem.

References

[1] Proxy discrimination — when an ostensibly neutral variable (here, postcode) correlates with a protected characteristic and carries its effect into a decision. Disagreement that clusters on such a variable is a standard signal that a category has absorbed a bias. ↩