Golden-Output Regression Test¶
Regression check / equivalence audit — instantiates Equivalence-Preserving Rewrite Optimization
Freezes the original form's outputs on a corpus of reference cases, then fails the rewrite if any output differs — treating recorded observable behavior as the equivalence oracle.
The Golden-Output Regression Test decides whether a rewrite preserved behaviour the simplest honest way possible: it records what the original form produced on a fixed corpus of inputs — the "golden" outputs — then runs the rewritten form on the identical inputs and fails on any difference. Equivalence here is not proved from the structure of the change; it is observed, byte for byte, against a frozen baseline. That is what makes it this mechanism and not its testing siblings: it needs a trusted reference version and exact comparison, and it is at its strongest exactly when outputs are deterministic and fully captured. Where a rewrite has an unambiguous "right answer" already on record, nothing catches unintended drift more cheaply or more decisively.
Example¶
A team is about to refactor a payroll-and-tax calculation engine for speed, and the one thing that must not move is the money. Before touching the code they capture outputs for ≈5,000 representative cases — a spread of filing situations, brackets, and deductions — and freeze them as the golden set. They refactor, then re-run all 5,000 through the new engine and diff against the frozen outputs.
The result: 4,997 match to the byte; three differ by a single rounding penny in a particular deduction edge case. That penny is the whole point. The rewrite reordered a rounding step, which is not an equivalence-preserving change for currency, and the golden set surfaced it before it reached a paycheck. The team either restores the original rounding order or, if the new order is genuinely correct, re-blesses those three references deliberately and on the record — never silently.
How it works¶
Three moves: capture the original's outputs on a fixed input corpus as the reference; re-run the rewritten form on the identical inputs; exact-compare and fail on any diff. The equivalence relation is therefore the recorded outputs themselves — it is observed, not derived. Because the comparison is exact, the test also has to draw the line between output that is observable behaviour and output that is incidental: timestamps, run IDs, or unordered collections get normalized or masked, and that masking decision is precisely where the observable-behaviour boundary is set. Too little masking makes the test brittle; too much lets real drift slip through.
Tuning parameters¶
- Corpus coverage — how many and which reference cases. A broader corpus catches more drift but costs more to capture and to run each time.
- Comparison strictness / masking — exact bytes versus normalized output. This dial is the observable-behaviour boundary: mask too aggressively and you hide real changes; mask too little and every intended cosmetic change floods the diff.
- Baseline trust — whether the golden outputs are known-correct or merely the current behaviour. It tests sameness against whatever you froze, bug and all.
- Refresh / re-bless policy — when the golden set is updated after an intended behaviour change. Stale goldens block legitimate work; careless re-blessing defeats the test.
- Diff granularity — whole-output versus field-level comparison. Field-level localizes exactly where a rewrite diverged.
When it helps, and when it misleads¶
Its strength is decisiveness and simplicity: when a trusted reference exists and outputs are exact, it catches any unintended change to observable output with near-zero false negatives and almost no cleverness required.
Its failure modes are the mirror image of that literalness. It is brittle — every intended change, however cosmetic, floods the diff and trains people to re-bless without looking. It tests sameness, not correctness: it enshrines the current output as the reference even if that output is wrong, which is why it is precisely a characterization test — a snapshot of what the system does, not a check of what it should do.[n1] It is blind to anything not in the captured output (side effects, timing, memory) and to any input not in the corpus. The classic misuse is re-blessing a failing golden set to make a rewrite "pass" without understanding the diff. The discipline: treat every diff as a question rather than a verdict, keep the corpus representative, mask only provably-incidental fields, and pair it with checks of correctness — not just of sameness.
How it implements the components¶
The test fills the archetype's checking components — the ones an exact-comparison oracle owns:
invariant_preservation_oracle— the recorded-output comparison is the oracle: identical outputs mean the rewrite preserved behaviour; any diff proves it did not.observable_behavior_boundary— the choice of which output fields are compared and which are masked as incidental draws the line around what counts as externally observable behaviour.
It does not produce the rewrite (Compiler Optimization Pass), does not measure cost (Benchmark Harness), and cannot handle the case where no trusted exact output exists — that is where the Metamorphic Test Suite checks preserved relations instead.
Related¶
- Instantiates: Equivalence-Preserving Rewrite Optimization — it is the oracle that certifies a rewrite left observable output unchanged.
- Sibling mechanisms: Metamorphic Test Suite · Property-Based Equivalence Test · Benchmark Harness · Compiler Optimization Pass
Editorial Notes¶
Form Classification¶
Form family: Experiment, Test & Rehearsal
Rationale: Golden-Output Regression Test operates as a bounded trial, probe, simulation, or rehearsal that generates evidence from performance because it freezes the original form's outputs on a corpus of reference cases, then fails the rewrite if any output differs — treating recorded observable behavior as the equivalence oracle.
Independent corroboration: The frozen evidence defines Golden-Output Regression Test as 'Freezes the original form's outputs on a corpus of reference cases, then fails the rewrite if any output differs — treating recorded observable behavior as the equivalence oracle', so its operative form is Experiment, Test & Rehearsal.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Computer Science & Software Engineering
Origin pattern: Single lineage
Present-day reach: Specialized
Rationale: Characterization testing canonically treats recorded observable output as a regression oracle.
Review outcome: Independent reviewer agreement; high confidence.
Notes¶
A green golden suite certifies sameness, not correctness: it says the rewritten form behaves identically to the baseline, which is exactly the guarantee equivalence-preserving rewriting needs — but it says nothing about whether the baseline was right to begin with. When the reference output may itself be wrong, this test must be paired with a check of what the output should be; on its own it will faithfully preserve a bug.
[n1] A test that pins down a system's current behaviour so later changes can be detected — a characterization test — captures what the code does, not what it ought to do. Golden-output testing is the canonical form: ideal for equivalence-preserving rewrites (where "unchanged" is the goal) and misleading if mistaken for a correctness check. ↩