Skip to content

Substitutability Trial / Canary

Production trial process — instantiates Representation-Independent Interface Contract

Proves a replacement in production by routing a slice of real traffic to it and promoting only if it behaves indistinguishably from the incumbent.

Offline tests can only check the inputs you thought to generate; production has inputs, load, and timing you did not. Substitutability Trial / Canary settles the substitution question empirically: it routes a slice of real traffic to the replacement implementation, runs it beside the incumbent, and promotes it only if it behaves indistinguishably — same results, no worse latency, no new errors — rolling back the instant it does not. Its defining move is deciding acceptance from live evidence under real load rather than from a specification or a test oracle: the replacement earns its place by surviving production at small scale before it is trusted at full scale. It is substitutability proven in the field, with a rollback as the safety net.

Example

A search service is replacing its storage backend — same query interface, entirely new engine underneath. Every offline conformance and differential test passes, but the team has been burned before by behaviour that only appears under production load. So they canary: 1% of live queries are mirrored to the new backend, and a comparator watches two things — do the results match the incumbent, and are latency and error rate within an agreed band?

For a day it looks clean. Then at 5% the comparator catches the new engine returning subtly different result ordering for queries with many tied-relevance hits, plus a p99 latency roughly 40% worse under a concurrency the offline tests never generated. Both breach the acceptance rule's thresholds, so the canary auto-rolls-back to 0% and files the divergence. No user outside the 5% ever saw it. The offline tests said "equivalent"; the canary said "not yet" — and it was right, because it tested the one thing they could not: the real thing.

How it works

  • Split traffic. Route a small, controllable fraction to the candidate — mirrored (shadow) or live — while the incumbent keeps serving.
  • Compare on live evidence. A comparator checks the candidate against the incumbent on real requests: same observable result, plus latency, error rate, and resource use within an agreed band.
  • Gate on an acceptance rule. Promote to a larger fraction only if the deltas stay inside threshold; breach it and roll back automatically.
  • Ramp, then commit. Widen the slice in stages (1% → 5% → 50% → 100%), each stage buying more evidence, until the candidate is either the new incumbent or rejected.

Tuning parameters

  • Traffic fraction and ramp schedule — how much traffic, escalated how fast. A slow ramp limits blast radius and buys evidence; a fast one reaches confidence sooner and risks more.
  • Acceptance thresholds — how large a divergence in results, latency, or error rate trips a rollback. Tight thresholds catch subtle regressions and cause more false rollbacks; loose ones let real ones through.
  • Shadow vs. live — mirror traffic to the candidate without serving its responses (safe, but can't measure user-facing effects), or actually serve a fraction (real, but exposes those users).
  • Rollback automation — whether a breach auto-reverts or pages a human. Automatic is faster and safer under load; manual retains judgement when the metric itself might be lying.
  • Comparison strictness — exact result-match versus tolerance for benign representation differences (ordering, formatting) — the same equality-relation dial a differential test faces, now on live data.

When it helps, and when it misleads

Its strength is testing the untestable-offline: real inputs, real concurrency, real resource contention, and the long tail of production the best generator never reproduces. Because it ramps behind a rollback, it converts a risky cutover into a reversible experiment — and it is the only mechanism that measures a substitute's performance cost of substitution, not just its correctness.[1]

Its failure modes are about the trial not seeing the failure. A canary too small or too short never routes the input that breaks, and green metrics then certify a substitute that is not equivalent — absence of evidence read as evidence of absence. Metrics that capture latency and errors but not semantics miss a contract violation entirely: the dashboards stay green while results are subtly wrong. The classic misuse is deciding to ship first and canarying as theatre — or setting the acceptance thresholds after seeing the results, so the trial can only pass. And a canary exposes real users to a possibly-worse implementation, a cost offline tests do not carry. The discipline is to define the acceptance rule and its thresholds before the trial, include a semantic comparator alongside the operational one, and treat the canary as the last gate after offline conformance — never a replacement for it.

How it implements the components

Substitutability Trial / Canary realises the acceptance side of the archetype — it makes the go/no-go substitution call on production evidence:

  • substitutability_acceptance_rule — its promote-or-rollback threshold is this rule, applied to live traffic: the candidate is accepted only if no client can tell it from the incumbent.
  • performance_and_resource_caveat — it is the mechanism that measures the cost of substitution — latency, error rate, resource use under real load — and folds those caveats into the accept/reject decision.

It does not establish conformance offline — that is Reference-Implementation Differential Test and Property-Based Conformance Test — and it does not record the version bump a successful promotion earns, which belongs to Semantic Versioning & Deprecation Gate.

Notes

The live comparator at a canary's heart is a differential test whose inputs are production traffic instead of a generator — so a team that already has Reference-Implementation Differential Test can often reuse its equality relation here. The canary adds what the offline test cannot: real load, and a rollback.

References

[1] The Liskov Substitution Principle is the formal statement of what a canary tests empirically — a replacement is acceptable only if no client can tell it apart from what it replaced. The canary swaps the theorem-prover for production traffic. ↩