Golden Case Benchmark Set¶
Benchmark suite — instantiates Equivalence-Relation Refinement and Coarsening
A frozen set of hand-curated anchor cases with their correct classifications, re-run after every relation change to prove that the cases which must stay stable still land where they should.
Every split or merge risks knocking loose something that was supposed to hold still. A Golden Case Benchmark Set is the regression net that catches it: a deliberately frozen collection of specimen cases, each paired with the class it is known to belong to, that gets re-classified under any proposed new relation and compared against its recorded answer. Its defining move is that it protects individual cases against unintended drift — it does not measure aggregate error rates and it does not map old classes to new; it asserts "these particular cases must always come out this way," and any change that moves one is stopped for review. It deliberately over-weights the hard specimens — the edge cases sitting near a boundary — because those are exactly where a well-meant refinement quietly reclassifies something it shouldn't.
Example¶
A law firm runs an automated document-sensitivity classifier that tags every file Public, Internal, Confidential, or Restricted, driving who can open it. The firm wants to refine "Confidential" into "Confidential" and a new "Client-Privileged" tier. Before flipping the relation, the records team runs the golden set: forty hand-picked documents whose correct tier was settled by counsel, chosen precisely because they sit on hard edges — a board deck that looks internal but contains a privileged legal memo, a marketing draft that quotes a sealed settlement. Under the proposed new relation, thirty-eight land exactly where they should. But two of the frozen cases — documents that must remain Restricted — slip into the new "Client-Privileged" tier, which grants broader access. That's a stop signal: the refinement, as drafted, would have widened access to two documents that counsel had locked down. The team fixes the split rule and re-runs until every anchor holds. The benchmark didn't decide the new relation; it guarded the promises the old one had made.
How it works¶
The golden set's distinguishing method is a frozen expected-answer contract. Each case is curated once, with its correct class fixed by an authority (a domain expert, a legal ruling, a prior adjudication) and a note on why it belongs there. The set is then held constant while the relation changes around it — the opposite of a data sample that refreshes. On each candidate revision the whole set is re-classified and diffed against the recorded answers; any case that moves is a flagged regression that must be explained (an intended reclassification is re-blessed and its expected answer updated deliberately; an unintended one blocks the change). Because the curated cases cluster on boundaries where classifications are fragile, the set is disproportionately sensitive to exactly the kind of quiet erosion a coarsening or refinement can cause.
Tuning parameters¶
- Set composition — how heavily to load edge and boundary cases versus clear central examples. Edge-heavy sets catch subtle drift but can over-alarm on genuinely intended boundary moves; central cases confirm the obvious cheaply.
- Freeze discipline — how hard it is to update an expected answer. Strict freezing catches every drift but requires ceremony to re-bless intended changes; loose freezing risks the benchmark quietly tracking the very drift it should detect.
- Coverage breadth — how many classes and contexts the anchors span. Broad coverage catches distant regressions; narrow sets are cheap but blind outside their neighborhood.
- Fail policy — whether any moved anchor blocks the change or only a weighted count does. Hard-blocking is safest for high-stakes relations; a budget tolerates churn where stakes are low.
- Provenance depth — how much of each anchor's why is recorded, so a flagged move can be adjudicated rather than merely noticed.
When it helps, and when it misleads¶
Its strength is turning "did we break anything important?" from a hope into a check that runs on every revision. It is the mechanism that stops erasure at the level of the specific case — the promise that this one document, this one patient, this one edge specimen still classifies the way it must — which aggregate error metrics can average right past. The technique is the classification cousin of golden master testing: pin the known-good outputs and alarm when they change.[n1]
It misleads when the set ossifies against a change that is actually correct — if the relation should now treat a case differently and the benchmark simply blocks it, the frozen answers become a brake on legitimate improvement. It also gives false comfort when coverage is narrow: passing forty anchors says nothing about the thousands of cases none of them resemble, so a benchmark can be green while a whole region silently drifts. And a benchmark that is quietly re-blessed every time it fails has stopped testing anything. The discipline is to treat a moved anchor as a question — intended or not? — to update expected answers only by deliberate re-blessing with a recorded reason, and to grow coverage as new regions of the relation come into play.
How it implements the components¶
The golden set fills the case-level continuity components — the ones a frozen regression suite can own:
boundary_case_sample_set— the curated anchors are precisely the edge and boundary specimens that must survive a relation change, held as a standing sample rather than rediscovered each time.invariant_preservation_check— re-running the set on each candidate revision checks that the property "these cases classify thus" is preserved, blocking changes that would silently violate it.
It does not hold the old-to-new class map — old_new_class_mapping — that is the Equivalence Class Crosswalk Table; the golden set only checks that specific anchors still land right, it does not translate whole vocabularies. And it does not adjudicate live ambiguous cases — that human work belongs to the Boundary Case Review Panel; the golden set is automated and frozen, not deliberative.
Related¶
- Instantiates: Equivalence-Relation Refinement and Coarsening — guards the invariant that designated cases stay classified as they must across relation changes.
- Sibling mechanisms: Equivalence Class Crosswalk Table · Classification Confusion or Error Matrix · Boundary Case Review Panel · Periodic Relation Granularity Review · Granularity Tuning Rubric
Editorial Notes¶
Form Classification¶
Form family: Experiment, Test & Rehearsal
Rationale: After every relation change, the frozen anchor cases are actively rerun to generate evidence that classifications expected to remain stable still do so.
Nearest alternative: Representation, Specification & Plan — The benchmark set is a durable artifact, but its operative use is repeated conformance testing.
Review outcome: Adjudicated after independent review; high confidence.
Origin Attribution¶
Primary origin: Computer Science & Software Engineering
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Multi-domain
Rationale: Characterization testing supplies frozen anchor cases that detect unintended behavior change.
Related originating lineages:
- Data Science & Analytics — Classification benchmark practice materially contributes hand-curated correct labels and drift checks.
Review resolution: Both reviewers agree that computer_science is primary: Characterization testing supplies frozen anchor cases that detect unintended behavior change. I retain data_science only as formative lineage, not as a list of later applications. I resolve origin_mode as cross_disciplinary_synthesis because the artifact joins distinct disciplinary contributions. I resolve domain_reach as multi_domain because it transfers across several fields but is not a domain-free primitive. Encyclopedia synthesis is true because the exact generalized packaging is an encyclopedia-authored combination or refinement.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
[n1] Golden master (a.k.a. characterization) testing pins a set of known-good outputs and flags any later change to them, letting a system be safely refactored by making unintended behavior changes visible. A golden case benchmark applies the same idea to a classification: the "master" is the recorded correct class for each anchor, and a relation change is the refactor being guarded. ↩