Side-Channel Regression Test¶

Continuous regression monitor — instantiates Side-Channel Leakage Containment

An automated suite that re-runs on every change to confirm previously-closed side channels stay closed — comparing observable behavior across matched secret-pairs and failing the build when they start to diverge.

A side channel that was closed can silently re-open: a refactor restores an early exit, a library upgrade defeats a constant-time routine, a new field re-introduces a size difference. Side-Channel Regression Test is the continuous gate against exactly that. It maintains a suite that, for each pair of protected states that are supposed to look identical, measures the observable — timing, size, error, resource use — and fails when the two become distinguishable beyond the allowed budget. Its defining move is that it runs repeatedly, tied to change, as a regression guard: where a one-shot differential test proves a channel closed once, this suite is what keeps it closed release after release.

Example¶

A cryptographic library's continuous-integration pipeline carries a side-channel regression suite. A previous release fixed a timing leak in signature verification, and the property is now encoded as matched pairs of inputs that must be indistinguishable on the timing channel: a valid versus an invalid signature, a secret key bit of 0 versus 1. On every commit the suite runs each pair many thousands of times and compares the timing distributions; if the distributions for the two cases diverge beyond the threshold, the build fails. This kind of statistical constant-time check compares timing across secret-dependent inputs precisely to detect such divergence.

Months later a maintainer "optimizes" the verification inner loop and, without realizing it, reintroduces a branch whose duration depends on a secret byte. The next CI run shows the two distributions pulling apart, and the build goes red before the regression can ship. The leak is caught not by an audit six months later but by the gate on the pull request that caused it.

How it works¶

Freeze the indistinguishability pairs. Turn each closed channel into matched secret-state pairs that must produce the same observable — the test oracles.
Fix the pass/fail bound. Set the distinguishability threshold the two members of a pair may not exceed (often a statistical distance between distributions).
Measure on every change. Re-run the suite per commit, sampling the observable enough times to detect a leak of the size you care about.
Fail loud, tie to the change. Break the build on divergence so the regression is pinned to the commit that introduced it.

Tuning parameters¶

Pair coverage — how many secret-distinguishing pairs the suite encodes. More pairs catch more channels but slow the pipeline.
Samples per run — trials per pair. More samples detect smaller leaks but cost CI time; this dial sets the smallest leak you can see.
Distinguishability threshold — how far two distributions may diverge before failing. Tighter catches subtler leaks but risks flaky failures from measurement noise.
Channels instrumented — timing, size, error, resource use. Each added channel widens coverage and lengthens the run.
Trigger cadence — per-commit (fast feedback, more cost) vs. nightly (cheaper, later catch).

When it helps, and when it misleads¶

Its strength is turning "we fixed it" into "it stays fixed": it catches a reintroduced leak at the moment it lands, automatically and cheaply, instead of during the next audit or after an incident. For properties like constant-time execution that regress silently under ordinary maintenance, it is the only thing standing between a fix and its quiet undoing.

Its limits follow from being a regression test. It only checks channels someone thought to encode as pairs — a genuinely novel channel is simply not in the suite, so a green build is not a proof of safety, only of no known leak reopening. Timing measurements are noisy in shared CI environments, producing flaky failures. And the classic misuse is to answer that flakiness by loosening the threshold until the test stops failing — silencing the alarm rather than fixing the leak or the measurement. This is the side-channel analogue of ordinary regression testing: you test what you fixed so it cannot quietly come back.^[1] The discipline that keeps it honest is to derive the pairs from the channel inventory, and to treat a flaky timing test as a measurement problem to fix rather than a nuisance to mute.

How it implements the components¶

This monitor fills the testing components — detection over time, not remediation:

red_team_side_channel_test_suite — the maintained, re-runnable suite of side-channel checks wired into the change pipeline.
protected_alternative_pair_set — the matched pairs of protected states that must stay observationally indistinguishable, used as the suite's oracles.

It detects but never remediates. The fixes it guards are built by Response Padding or Coarsening and Secret-Independent Resource Scheduling; one-shot exploratory measurement of a suspected new channel is Differential Observation Test's role, not this suite's continuous regression check.

Instantiates: Side-Channel Leakage Containment — keeps closed channels closed as the system changes.
Consumes: Side-Channel Inventory Workshop supplies the channels and protected states to encode as test pairs.
Sibling mechanisms: Secret-Independent Resource Scheduling · Differential Observation Test · Response Padding or Coarsening · Side-Channel Inventory Workshop · Residual Leakage Review Board · Query Rate and Composition Limit · Threshold Suppression · Privacy-Preserving Telemetry View · Batching and Delayed Release · Broker Visibility Partitioning · Cache Partitioning or Flush Rule · Constant Response Envelope · Controlled Noise Injection · Error Message Normalization · Metadata Minimization Filter

Notes¶

Differential Observation Test and this suite are complementary and easily confused: the differential test is a one-shot instrument for discovering or confirming whether a channel distinguishes two states; the regression test is the standing guard that re-checks known channels on every change. The suite is also the review board's evidence source — a green history is what the board weighs when accepting residual risk.

References¶

[1] A regression test re-verifies that a defect, once fixed, does not return as the code evolves. Side-channel regression testing applies the same idea to an observable property (constant time, constant size) rather than a functional result — the failure it guards against is a previously-closed leak silently reopening. ↩