Skip to content

Strict-Mode Shadow Run

Pre-enforcement rehearsal method — instantiates Asymmetric Interface Tolerance Calibration

Runs a stricter rule set in log-only mode against live traffic to count exactly what it would reject — before any of it actually blocks — so tightening is a measured step, not a gamble.

Tightening a tolerant interface is frightening because nobody knows how much real traffic the stricter rule will break. Strict-Mode Shadow Run removes the guesswork: it evaluates the proposed strict rules against live production traffic in log-only mode — recording what would have been rejected while actually rejecting nothing[1]. Its defining move is to separate measuring a tightening from enforcing it: the strict rule runs for real, on real inputs, but with its teeth retracted, so the blast radius becomes a number you can read before you ever flip enforcement on. It is the dress rehearsal that turns "we think this is safe to tighten" into "≈3% of yesterday's traffic would have failed, almost all from two partners."

Example

A transit agency wants to stop accepting a sloppy legacy barcode format at its fare gates and require the new spec. Flipping that on blind risks stranding thousands of riders mid-commute. So the new, stricter validator is deployed in shadow: at each tap it runs alongside the live-tolerant validator, and while the gate still opens on the old rules, it logs whether the strict rule would have rejected the barcode, and why. A week of real taps produces the picture — say ≈4% would fail, concentrated in one transit-card vendor's older batch and in phone-screen glare, not spread evenly across riders. That telemetry redirects the work: fix the one vendor and add screen-glare tolerance before enforcing, instead of discovering both at the gate with a line forming behind an angry commuter.

How it works

  • Two validators, one decision. The current lenient rule stays authoritative and controls what actually happens; the strict candidate runs in parallel purely to record its verdict.
  • Log the counterfactual. For each live input the shadow rule emits "would-accept / would-reject + reason," building a real-traffic map of exactly what tightening would cost and where it lands.
  • Read the distribution, not just the rate. The value is in who and what the rejections concentrate on — a few producers, one edge case — which tells you what to fix before enforcing.
  • Promote only when the number is acceptable. The shadow verdict becomes the go/no-go for switching enforcement on, and the same harness re-runs to confirm the fixes worked.

Tuning parameters

  • Shadow duration and coverage — how long and across how much traffic the shadow runs. Longer captures rare and seasonal cases but delays the tightening; too short undercounts the tail.
  • Sampling rate — whether every input or a sample is shadow-evaluated. Full coverage is complete but costly; sampling is cheap but blurs rare failures.
  • Breakdown dimensions — how finely would-reject events are attributed (by producer, version, field, path). Richer attribution turns a scary aggregate into a targeted fix list.
  • Promotion threshold — the would-reject rate (and its concentration) you'll tolerate before flipping enforcement on. Encodes how much residual breakage is acceptable.
  • Representativeness controls — whether shadow traffic is checked for seasonality and edge coverage, so the measured rate isn't a quiet undercount.

When it helps, and when it misleads

Its strength is that it makes tightening empirical: instead of debating whether a stricter rule is safe, you measure it on the exact traffic it will face, and you get a prioritized list of what to fix and whom to warn before a single request is blocked. It de-risks the scariest step on the path from tolerant to strict, and it turns a political argument into a shared number.

It misleads chiefly through unrepresentative shadow traffic. A run that misses month-end batches, a seasonal partner, or a rare device will report a reassuring rate that enforcement then exceeds — the measurement was honest but the sample wasn't. Its classic misuse is running the shadow, getting an inconvenient number, and shipping enforcement anyway on schedule — treating the rehearsal as a formality rather than a gate. And a shadow run only tells you what would break, never whether the tightening is worth it; that judgment lives elsewhere. The discipline is to size the run to cover the real cycle, publish the breakdown to the affected producers, and let the number actually hold the launch.

How it implements the components

  • malformed_input_telemetry — its output is telemetry: a real-traffic, attributed record of what the stricter rule would reject and why, gathered without disrupting anyone.
  • tightening_and_deprecation_path — it is the measure-before-you-enforce step on that path, gating each tightening on evidence rather than nerve and confirming fixes before promotion.

It does not define the strict rule it rehearses (that comes from Strict Output Linter or Strict Bidirectional Contract Gate), does not schedule and announce the deprecation to partners (Deprecation Warning and Tightening Schedule), and does not chart ongoing malformation rates as a standing view (Malformation Rate Dashboard).

References

[1] Dark launch / shadow traffic — exercising a new code path on real production traffic while its results are recorded but not acted on. It is the standard way to measure a change's real-world impact before it can affect anyone, and here it is what lets a tightening be sized before it blocks. ↩