Skip to content

Backward Compatibility Test

A backward-compatibility check — instantiates Behavior-Preserving Refactoring

Checks that a new version still honors every promise existing consumers already rely on, so an internal change can ship without breaking anyone downstream.

A Backward Compatibility Test asks one directional question: does the new version still keep faith with what existing consumers depend on? Its defining feature is asymmetry — it treats the old contract as the fixed point and checks that the new version still satisfies it, not that the two are identical. New behavior may add (extra fields, new endpoints, newly accepted inputs) but must not remove, rename, or tighten what was already promised. It is scoped to the external contract surface — the interfaces, protocols, formats, error codes, and guarantees others build against — and it is driven from the consumer's side, so the pass condition is "an existing client still gets what it expected," not "the internals look the same."

Example

A team refactors the implementation behind a payments API and, in passing, wants to add a currency field to the response. Existing mobile apps — some versions years old — parse that response in the field. The Backward Compatibility Test replays recorded request/response interactions from those old client versions against the new build and asserts, from each client's vantage: every field it reads is still present, same type, same meaning; the new currency field is additive and safely ignorable; error codes are unchanged; and the idempotency and ordering guarantees still hold. It passes when a two-year-old client, reading exactly the fields it always read, still gets a valid response — and it fails loudly the moment the refactor accidentally renames a field or tightens a validation that an old client would trip on.

How it works

  • Enumerate the promises, not the whole behavior. Pin down only the externally relied-upon surface — fields, types, error codes, protocol semantics — both documented and de-facto.
  • Drive from the consumer's side. Replay real or representative old-client interactions and assert from their point of view.
  • Apply the asymmetry rule. Additions pass; removals, renames, and tightenings fail. The bar is "still satisfies the old contract," not "identical output."
  • Cover de-facto dependencies. Include undocumented-but-relied-upon behaviors, not just the written spec.

Tuning parameters

  • Contract-surface scope — how much counts as "the promise": the documented API only, or every observable including timing and ordering. Wider catches more breakage but flags more churn.
  • Consumer-version horizon — how far back support extends. Longer horizons protect stragglers but constrain what the new version may change.
  • Asymmetry strictness — whether additive changes are always safe or themselves reviewed, since some brittle clients break on unexpected fields.
  • Evidence source — synthetic contract cases versus replayed real traffic from old clients. Real traffic captures de-facto dependencies but needs capture infrastructure.

When it helps, and when it misleads

Its strength is turning "will this break someone?" into a checkable question — it catches the silent-rename and tightened-validation class of breakage, and its asymmetric framing matches how real deprecation actually works.

Its limit is that you can only test the dependencies you know about. With enough consumers, every observable behavior ends up depended on by someone regardless of what the contract says, so a green backward-compatibility test bounds the known contract, never proves that nobody breaks.[1] The classic misuse is narrowing the contract surface until the test passes — declaring an inconvenient dependency "not really part of the contract" after the fact, which defines the breakage away instead of handling it. The discipline is to derive the contract surface from real consumer usage rather than from what is convenient to preserve, and to pair any removal with a deprecation window instead of a silent drop.

How it implements the components

The consumer-facing check side of the archetype — the promise-keeping components:

  • compatibility_check — it is the check that new-versus-old conformance holds for existing consumers.
  • interface_contract — it verifies the interface and protocol signature those consumers call against.
  • external_relation_contract — it upholds the broader promise to downstream parties: formats, semantics, and guarantees beyond the raw signature.

It checks one directional promise, not a standing battery: the behavior_preservation_test, regression_guard, and tolerance_band of the full matrix are Compatibility Test Suite, and the reference cases it asserts against — the golden_case_library and legacy_behavior_baseline — come from the Golden Case Benchmark.

Notes

"Backward compatible" is not "identical." A new version may add fields, endpoints, or accepted inputs and still be perfectly compatible; a test calibrated to demand byte-identical output will block safe additive changes and slow legitimate evolution. Calibrate the assertions to the old contract's guarantees, not to exact equality — that is what separates this from a plain behavior-diff.

References

[1] Hyrum's Law — with a sufficient number of users, every observable behavior of a system will be depended on by someone, regardless of what the contract promises. Used correctly here: a passing test bounds known dependencies, not all of them. ↩