Skip to content

Golden Case Benchmark

A reference-behavior artifact — instantiates Behavior-Preserving Refactoring

A curated library of canonical input-to-output cases, captured from the current system, that serves as the fixed reference for judging whether a refactor changed observable behavior.

A Golden Case Benchmark is the frozen reference truth a refactor is measured against — a hand-curated set of canonical cases, each paired with the output the current system produces for it, checked in and treated as the definition of "unchanged." Its defining move is to record today's behavior as the standard: it makes no claim that the current outputs are correct in the abstract, only that a behavior-preserving change must reproduce them. Because it captures behavior as concrete, human-readable exemplars through the public surface — not as internal assertions — the benchmark outlives the code it came from and survives even a large internal rewrite. It is deliberately inert: it holds the reference data, not the machinery that runs or gates it.

Example

A team maintains a twenty-year-old payroll engine and wants to restructure its tangled tax module. Before touching anything, they build a Golden Case Benchmark: roughly 300 canonical employee scenarios chosen for coverage — hourly and salaried, multi-state, garnishments, mid-year raises, and edge cases like a $0 net and a negative retroactive adjustment. They run every scenario through the current engine and record each resulting paycheck line as the golden output. That recorded set — inputs plus current outputs — becomes the fixed reference. It doesn't assert the numbers are right; it asserts "this is what we produce today, and the refactor must reproduce it exactly, save for the handful of cases we've flagged as known-wrong." Later, after restructuring, they replay the 300 cases and diff against the golden file. The outcome is what turns "I think it still works" into "294 cases match to the cent, 6 differ, and here is precisely which lines moved."

How it works

  • Curate for coverage, not volume. Pick canonical cases spanning the behavior's important regions — typical, boundary, and historically buggy — rather than piling up near-duplicates.
  • Record current output as the reference. Define "same" as "same as today" by capturing a locked snapshot of the current system's outputs.
  • Freeze and version. The golden file is checked in and changes only by a deliberate, reviewed re-blessing, so drift can't happen silently.
  • Stay behavior-level. Reference behavior through the public surface so the benchmark keeps working across arbitrary internal restructuring.

Tuning parameters

  • Case-selection breadth — how many and how varied the canonical cases. Wider coverage catches more drift but costs curation and slows every replay.
  • Fidelity of recorded output — full byte-exact capture versus selected fields. Exact capture catches subtle drift but is brittle to irrelevant formatting churn.
  • Re-blessing policy — how hard it is to update a golden value. Strict prevents silent redefinition of "correct"; loose speeds legitimate change but invites rot.
  • Provenance tagging — whether each value is marked verified correct versus merely current behavior, so bugs-to-preserve are distinguished from truth.
  • Refresh scope — regenerate the whole library or only touched cases when behavior legitimately changes.

When it helps, and when it misleads

Its strength is giving a refactor a concrete, inspectable definition of "unchanged" that a human can read case by case — it catches drift the author never thought to assert, and because it lives at the behavior level it survives rewrites that shatter unit-level tests.

Its central trap is that it blesses current behavior including current bugs, so it can quietly lock in defects that someone downstream may already depend on.[1] Coverage gaps are invisible: a green benchmark over a thin case set is false comfort, guarding only the regions you thought to include. And the classic misuse is re-blessing a failing golden value to make the suite pass instead of investigating the diff — redefining "correct" to match the new code. The discipline is to treat every diff as guilty until explained and to re-bless only through review with a recorded reason.

How it implements the components

The reference-artifact core of the archetype — the components an inert benchmark actually holds:

  • golden_case_library — the benchmark is the curated library of canonical cases; its primary substance.
  • legacy_behavior_baseline — recording the current system's outputs fixes the baseline snapshot the refactor must reproduce.
  • observable_behavior_contract — the golden cases are the concrete, executable statement of what the outside is entitled to observe.

It is the reference data, not the check: the behavior_preservation_test, regression_guard, and tolerance_band that run and judge the cases are Compatibility Test Suite and the Behavioral Diff Gate, and the compatibility_check against existing consumers is the Backward Compatibility Test.

Notes

The benchmark has no opinion on how much deviation is tolerable or when to fail a build — that judgment lives in the tolerance band and the diff gate that read it. Keeping the reference data separate from the pass/fail logic is what lets you grow coverage without touching gate behavior, and retune the gate without disturbing the cases. It also differs from an automatically captured characterization harness: this library is hand-picked and human-meaningful, chosen so a reviewer can read a case and know what it protects.

References

[1] Golden master (a.k.a. approval) testing — capture a known output, then assert future runs reproduce it. Used correctly here: it pins current behavior, which is exactly what a behavior-preserving change must reproduce, bugs and all. ↩