Classification Audit¶
Diagnostic audit — instantiates Schema Update Protocol
Samples already-classified items and re-judges them to measure how often the current schema misfits, producing the violating cases and named mismatch that justify a revision.
A Classification Audit is a measurement instrument, not an editor: it takes items the schema has already sorted, re-judges a sample of them against the schema's own rules, and reports how often the label was wrong, forced, or contested. Its defining move is that it changes nothing about the schema — it only quantifies misfit and hands back evidence. That restraint is the point. A schema update that begins with "someone felt the categories were off" is churn waiting to happen; an update that begins with "in a 400-item sample, 18% of items tagged X were re-judged as not-X, and the disagreements cluster on one boundary" has a rate, a pattern, and a named gap. The audit turns diffuse dissatisfaction into a measured, reproducible signal that the rest of the protocol can act on or dismiss.
Example¶
A trust-and-safety team runs a content-moderation schema with a label called harassment. Moderators keep muttering that it feels overloaded, but muttering is not evidence. The team runs a Classification Audit: they pull a stratified sample of 500 recently-labeled posts, have two senior reviewers independently re-label each one blind to the original decision, and compare. The headline number is stark — original and re-judged labels agree only 71% of the time on harassment, versus 94% on spam. More useful than the number is where the disagreement lives: nearly every conflict is a post that is hostile but aimed at a public figure's argument rather than a person, which reviewers split on because the schema gives them one bucket for both. The audit writes this up as a named mismatch ("harassment conflates targeted abuse with heated public criticism"), attaches the fifty clearest split cases as an exhibit, and records the disagreement rate as a baseline. It stops there. It does not propose splitting the category or rename anything — it has proven a problem exists and sized it, which is all a diagnostic is allowed to do.
How it works¶
What distinguishes an audit from ordinary complaint-gathering is that it is sampled and re-judged, so its output is a rate rather than an anecdote:
- Draw a sample with structure. Stratify by category (and by volume, recency, or risk) so rare-but-costly categories are not swamped by common ones.
- Re-judge blind. At least two judges re-classify each item without seeing the original label; agreement is measured, not assumed.
- Localize the disagreement. The valuable output is not the overall error rate but where the errors cluster — which boundary, which pair of categories, which kind of item — because that is what names the mismatch.
- Baseline and repeat. The same audit re-run after a revision is the error monitor: it says whether the fix reduced misfit or merely moved it.
Tuning parameters¶
- Sample size and stratification — bigger and finer samples tighten the error estimate and expose rare failures, but cost reviewer time; match to how consequential the categories are.
- Number and seniority of judges — more independent judges make the disagreement signal trustworthy but multiply cost; a single re-judge only detects gross error.
- Agreement metric — raw percent agreement is legible but flattered by common categories; a chance-corrected measure is fairer but harder to explain to stakeholders.
- Blinding rigor — fully blind re-judging is cleanest; showing the original label is cheaper but invites anchoring.
- Cadence — a one-off diagnostic versus a standing periodic audit that doubles as ongoing monitoring.
When it helps, and when it misleads¶
Its strength is that it disciplines the whole protocol's entry gate: it makes revision earn its way in with a measured misfit rate and a localized pattern, and the same instrument re-run later tells you honestly whether a change worked. The right metric here is inter-rater reliability — the degree to which independent judges applying the same schema reach the same label — because low reliability is the sharpest evidence that the fault is in the schema and not the judges.[n1]
Its failure mode is mistaking measurement noise for structural mismatch. If judges are undertrained or the sample is tiny, low agreement reflects sloppy application, not a broken schema — and "fixing" the schema then chases a phantom. A classic misuse is running the audit only where you already suspect trouble, so the sample is not representative and the error rate is manufactured. The guarding discipline is to separate the two hypotheses before concluding: re-audit a well-understood, stable category as a control, and only call it a schema mismatch when disagreement concentrates on a specific boundary rather than smearing evenly across everything — even, and unglamorous, error is usually a training problem.
How it implements the components¶
violating_case_set— the audit's exhibits: the concrete re-judged items that did not fit cleanly, retained as evidence for whatever revision follows.schema_mismatch— the audit's written verdict names the gap in the schema's own terms (which boundary, which conflation), turning a rate into an actionable problem.error_monitor— re-run against a post-revision baseline, the same procedure measures whether misfit fell without new confusion appearing.
A Classification Audit deliberately stops at diagnosis: it does not perform revision_decision (that is Category Split/Merge Review) or category_revision (that is Taxonomy Revision). Its nearest twin is Schema Migration Workflow, which also monitors — but that page watches a single migration run for dropped or mis-mapped records at cutover, whereas this audits a live schema for standing misclassification.
Related¶
- Instantiates: Schema Update Protocol — the audit is the evidence-gathering front of the loop that keeps revision tied to measured misfit.
- Sibling mechanisms: Category Split/Merge Review · Coding-Frame Revision · Glossary Update · Knowledge-Base Retagging · Ontology Refactoring · Schema Change Review Board · Schema Migration Workflow · Taxonomy Revision
Editorial Notes¶
Form Classification¶
Form family: Assessment, Review & Assurance
Rationale: Samples already-classified items and re-judges them to measure how often the current schema misfits, producing the violating cases and named mismatch that justify a revision, making its operative form a bounded evaluation of existing evidence or work that produces a finding or disposition.
Independent corroboration: The frozen evidence defines Classification Audit as 'Samples already-classified items and re-judges them to measure how often the current schema misfits, producing the violating cases and named mismatch that justify a revision', so its operative form is Assessment, Review & Assurance.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Library & Information Science
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Multi-domain
Rationale: Taxonomy and cataloging practice supplied governed schemas and expert reassessment of whether assigned classes fit their definitions.
Related originating lineages:
- Data Science & Analytics — Classification evaluation contributes sampled error analysis and systematic mismatch patterns.
- Statistics & Experimental Design — Audit sampling supplies reproducible rates and uncertainty rather than anecdotal complaints.
Review resolution: Taxonomy governance is the agreed primary lineage because the audit re-judges whether assigned classes fit controlled definitions. Data-science error analysis and statistical audit sampling produce reproducible boundary-specific rates; the diagnostic-only bundle is an Encyclopedia synthesis.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
[n1] Inter-rater reliability measures how much independent judges agree beyond chance when applying the same classification rules; a chance-corrected coefficient such as Cohen's kappa is the standard form. Persistently low reliability on one category, with high reliability on its neighbors, is the audit's cleanest evidence that the schema — not the judge — is at fault. ↩