Skip to content

Differential Observation Test

Diagnostic test — instantiates Side-Channel Leakage Containment

Feeds pairs of inputs that differ only in the protected value and measures whether their observable behavior is distinguishable — turning 'does it leak?' into a measurement.

You cannot contain a leak you cannot see, and most side channels are invisible to inspection. Differential Observation Test makes leakage measurable: it constructs pairs of scenarios that are identical except for the protected value, drives the system with each, and statistically compares what an observer could perceive — timing distribution, response size, error, resource use. Its defining move is the matched pair: by holding everything constant except the secret, it isolates the secret as the only possible cause of any observable difference, so a measured difference is a leak. It produces no fix — it detects and sizes leaks so the containment controls know whether they are working.

Example

A team suspects a signature-verification routine in a cryptographic library runs slightly faster or slower depending on a secret key bit — a timing leak. The test builds many input pairs that differ only in that bit, executes each pair thousands of times under a defined observation model (the observer can time individual calls), and compares the two timing distributions with a statistical test. If the distributions are distinguishable beyond measurement noise, the routine is not constant-time and the leak is real and sized; if not, it passes under that observation model and sample size. This is the logic of constant-time verification.[1]

How it works

  • Construct matched pairs. Generate scenarios identical but for the protected value, so the secret is the only variable.
  • Fix an observation model. State exactly what the assumed observer can measure and how many times, because the answer is only meaningful relative to that model.
  • Run and compare distributions. Collect the observable for each side of the pair and apply a statistical distinguisher, not a single-shot comparison.
  • Report distinguishability vs. threshold. Output a leakage measure against the acceptance line, so the result feeds directly into go/no-go and regression gating.

Tuning parameters

  • Pair-construction strategy — worst-case pairs (adversarially chosen) find leaks fastest; random pairs estimate typical exposure.
  • Adversary/observation model — what the observer can measure and how many repetitions; a stronger model makes the test more sensitive and the guarantee stronger.
  • Sample size / statistical power — more runs detect smaller leaks at higher cost; too few miss real ones.
  • Distinguisher and threshold — the test statistic and the pass line — the distinguishability budget the result is judged against.

When it helps, and when it misleads

Its strength is turning a vague worry into a number that can be tracked and regressed — the input every containment control needs to know whether it worked. Its central limitation is that a pass is scoped: "not distinguishable under this observation model at this sample size" is absence of evidence, not proof of safety, and a stronger attacker or a longer run can still separate the cases. The classic misuse is weakening the observation model until the test passes, or running it once and assuming permanence as the code drifts. The discipline is to fix the adversary model up front and freeze every passing configuration into a Side-Channel Regression Test so regressions are caught.[1]

How it implements the components

  • protected_alternative_pair_set — the matched, secret-differing scenario pairs are this test's defining artifact.
  • adversary_prior_and_query_model — the explicit model of what the observer can measure and how often, against which distinguishability is judged.

It measures leakage but applies no remedy: equalizing the response is Constant Response Envelope's and blurring with noise is Controlled Noise Injection's; and it is the exploratory measurement, not the CI harness — freezing a passing result into a regression suite is Side-Channel Regression Test's.

Notes

A green result is only as strong as the observation model fed in; the test's honesty depends on stating that model explicitly and choosing it to match — or exceed — the observer the system actually faces. Passing tests belong in the regression suite, not the archive.

References

[1] Constant-time verification — testing that a routine's observable behavior (typically execution time) does not depend on secret inputs, usually by statistically comparing timing distributions across secret-differing input classes. A pass certifies indistinguishability only under the tested observation model. ↩