Skip to content

Scenario Testing

Test protocol — instantiates Model-Based Regulation

Checks the regulator against a curated set of plausible, extreme, and boundary situations, asking of each: does it stay within safe limits and degrade gracefully?

Scenario Testing checks the regulator against a deliberately chosen set of situations — the plausible, the extreme, and the awkward boundary cases — and asks a pass/fail question of each: in this situation, does the policy keep the system within its safe limits, and when pushed past them, does it fail gracefully into a known safe state? Where the other tests sweep ranges or replay history, scenario testing is curated and discrete: a hand-built portfolio of named cases, each with an acceptance criterion decided in advance. Its purpose is not to estimate typical performance but to confront the policy with the specific situations someone is worried about and get a clear verdict on each.

Example

A reservoir's spillway gates are governed by an automated policy balancing flood risk downstream against water storage. Before a revised policy is trusted, engineers assemble a scenario portfolio: a routine spring melt (plausible), a probable-maximum-flood inflow arriving in a single night (extreme), a flood that crests exactly as a gate actuator jams (boundary/compound), and a sensor blackout during rising water (degraded-information). Each scenario carries a pre-agreed acceptance criterion — downstream flow must stay under the levee-design limit; storage must not exceed the dam's safety pool.

Run through the portfolio, the revised policy passes the routine and extreme-inflow cases but fails the jammed-gate scenario: it keeps commanding the stuck gate and lets the reservoir climb past its safety bound instead of dumping through the remaining gates. It also fails to drop into the pre-defined safe holding configuration when the sensors go dark. Those two red cells are the deliverable — not a score, but a precise list of the situations the policy does not yet survive — and they send it back for a fix aimed exactly there.

How it works

Scenario testing is defined by how its cases are chosen and judged, not by any engine that runs them:

  • Cases are selected for worry, not coverage. The portfolio deliberately spans plausible, extreme, and boundary/compound situations — especially the compound ones, two things going wrong at once, that ranges and averages miss.
  • Each case has a verdict decided in advance. Acceptance criteria — a safety limit not to breach, a fallback to reach — are fixed before the run, so the result is an unambiguous pass or fail rather than a number to interpret.
  • Failure into the fallback is itself a test. It checks not only that the policy stays safe when it can, but that it degrades into a known safe state when it cannot — surviving the situation, not just handling the happy path.
  • It borrows an execution environment. The scenarios are usually run on a simulator or digital twin; scenario testing supplies the cases and criteria, not the dynamics.

Tuning parameters

  • Scenario portfolio — which situations make the list and how extreme they go. A richer portfolio catches more but costs authoring effort, and can never be exhaustive.
  • Acceptance strictness — how hard the pass bar is per scenario. Strict bars catch subtle failures but can red-flag behaviour that is actually tolerable.
  • Compound depth — how many things go wrong at once in a scenario. Deeper compounds expose brittle interactions but explode the number of cases.
  • Plausible-to-extreme balance — how the portfolio is weighted between everyday and once-in-a-lifetime cases. Over-weighting extremes hardens against black swans but may neglect common failure modes.
  • Fallback-reach requirement — whether merely "staying safe" passes, or the policy must also reach the defined safe state. Encodes how much you trust graceful degradation versus prevention.

When it helps, and when it misleads

Its strength is that it gives a clear, per-situation verdict on exactly the cases stakeholders fear, and it is the test that most directly checks the two things regulation must never get wrong — breaching a safety bound and failing to reach a safe fallback.

Scenario testing can only ever cover the scenarios someone thought to write. It is blind to the situation nobody imagined, so a wall of green can breed false assurance against the case that actually arrives — the beyond-design-basis event.[1] It also invites teaching to the test: tuning the policy to pass the named scenarios while leaving the space between them fragile. The discipline is to treat the portfolio as a floor and not a ceiling, refresh it with every near-miss and post-incident finding, and pair it with range-sweeping and adversarial methods that probe the gaps a curated list leaves.

How it implements the components

Scenario Testing fills the safety-acceptance slice — the components that decide whether the regulator behaves acceptably in the situations that matter most:

  • safety_bound — each scenario operationalizes a safety limit into a concrete pass/fail check the policy must not breach.
  • safe_fallback_state — boundary and degraded scenarios verify that, when it cannot stay safe by control, the policy reaches the defined safe state rather than pressing on.

It supplies cases and criteria, not the machinery around them: the execution environment comes from Digital Twin Trial, the parameter-range robustness from Sensitivity Analysis, and the recorded history from Historical Replay.

Notes

Scenario Testing and a Model-Failure Red Team pull in opposite directions and need each other. Scenario testing confirms the policy survives the situations you listed; a red team's job is to invent the situation you didn't. A green scenario suite tells you the known hazards are covered — not that the unknown ones are — so the two together, rather than either alone, is what earns confidence.

References

[1] Engineering safety distinguishes design-basis events — the scenarios a system is explicitly built and tested to withstand — from beyond-design-basis events that exceed them. Scenario testing verifies the design-basis set; it says nothing about what lies beyond it, which is why a complete pass is necessary but never sufficient. ↩