Skip to content

Regression Test Suite

Behaviour-preservation monitor — instantiates Specification-to-Execution Lowering

Re-runs a corpus of previously-passing cases against each new version so that any unintended loss of working behaviour breaks the build, using the system's own recorded past output as the reference.

A Regression Test Suite guards against one specific danger in a lowering that keeps changing: that a new version silently loses behavior the old version had. It is a growing corpus of cases whose expected results were captured from a version already believed correct, re-run automatically on every change so that any deviation fails the build. Its defining trait is where its oracle comes from: unlike a contract or specification test, a regression suite does not check the system against an external statement of what it should do — it checks the system against its own recorded past. A case earns a place precisely because the behavior it pins was once working and must not regress. The suite's whole purpose is to make the loss of that behavior loud and immediate instead of a surprise discovered in production.

Example

A payroll engine computes net pay from gross across thousands of tax and deduction rules. Before a refactor that reorganizes the deduction logic, the team runs the current, trusted engine over a corpus of ~50,000 realistically-shaped employee scenarios and records every net-pay result as the golden reference. After the refactor, the regression suite replays the same inputs and diffs each result against its golden value. Most match; three do not — employees near a specific pension-contribution cap now compute a net pay a few cents lower.

Nothing in the requirements changed, so this is not a feature — it is a regression: the refactor dropped a rounding step. The build goes red, the diff points straight at the affected cases, and the change is blocked until the behavior is restored or the golden values are deliberately, reviewably updated. The suite never held an external definition of "correct net pay"; it only held what the trusted prior version produced, and it defended exactly that.

How it works

  • Capture, then defend. Expected results are snapshotted from a version deemed good; that snapshot becomes the oracle. New behavior is judged against the old, not against a spec.
  • Replay on every change. The corpus runs on each commit, release, or regeneration, so a lost behavior surfaces at the change that caused it.
  • A red result is a trigger. A failure is the operational signal to stop, roll back, or regenerate — not merely information to file.
  • Deliberate re-baselining. When a change is meant to alter output, the golden values are updated as a reviewed act tied to that version, never silently.

Tuning parameters

  • Golden-update discipline — how hard it is to bless new expected values. Loose updating makes the suite pass easily but lets real regressions be waved through as "expected changes"; strict updating catches more but slows intended changes.
  • Corpus selection — which cases earn a permanent guard. Broad corpora catch more but run slower and flag benign churn; curated corpora are fast but leave gaps.
  • Comparison strictness — exact match versus tolerance versus invariant. Exact match catches the smallest drift but breaks on cosmetic or nondeterministic differences.
  • Trigger severity — whether a failure blocks the build, warns, or opens a ticket. Blocking maximizes protection but can gate delivery on a single flaky case.
  • Snapshot cadence — how often the golden baseline is refreshed to the current version, trading currency of the reference against stability of the guard.

When it helps, and when it misleads

Its strength is making the loss of hard-won behavior impossible to ship silently — especially valuable when a lowering is refactored or regenerated repeatedly and no one can hold all prior behavior in their head. It turns "did we break anything?" from a hope into a checked fact for every case in the corpus.

Its limits are the limits of its corpus and its oracle. It can only defend behavior it has examples of — an untested path is an unguarded one, and a green suite proves no known behavior was lost, not the system is correct. Because the oracle is the system's own past, a bug present when the golden values were captured is enshrined as correct, and the suite loyally defends it. The classic misuse is blessing golden files without reading them — regenerating expected outputs to make a red suite green, which quietly certifies whatever the new code now does (approval testing degraded into rubber-stamping).[1] The discipline is to treat every golden update as a reviewed diff, and to pair the suite with spec-derived checks so the reference is not merely "what we used to do."

How it implements the components

Regression Test Suite fills the behavior-preservation and change-feedback components a monitor built on recorded history can fill:

  • reference_model_or_oracle — the golden results captured from a trusted version are the oracle each new version is judged against; here the oracle is the system's own past output, not a parallel implementation.
  • operational_feedback_and_regeneration_trigger — a failed case is the signal that fires: block, roll back, or regenerate the change that lost the behavior.
  • versioned_change_record — each guarded behavior is pinned to the version it was captured from, so a failure localizes to the change delta that broke it.

It does not decide what the behavior *should be from a specification (proof_and_test_obligation_set and declared-boundary cases belong to the Model Checker and Static Analyzer and to the Contract Test Suite), nor define equivalence between two independent implementations (correspondence_criterion — a Differential Equivalence Test); it defends continuity of one system's behavior across its own versions.*

Notes

The suite's oracle is only as good as the version it was captured from — it defends continuity, not correctness. Paired with a spec-derived check (Contract Test Suite) or an independent oracle (a Differential Equivalence Test), it stops enshrining old bugs as the standard to preserve.

References

[1] Approval (snapshot) testing — a test records a system's output as the approved baseline and fails on any later difference, so a human need only review the diff when it changes. Its well-known hazard is approving a diff without actually reading it, which launders a regression into the new baseline; disciplined review of every approved change is the standard guard. ↩