Sensitivity Analysis¶

Robustness assessment — instantiates Model-Based Regulation

Sweeps the model's inputs and parameters across their plausible ranges to find which ones actually move its decisions — and whether the model's added complexity earns its keep.

Sensitivity Analysis turns the dials on a model to see which ones actually move it. It sweeps the model's inputs and parameters across their plausible ranges and watches what the regulator's decision does in response: which handful of parameters swing the outcome, which are inert, and over what range the model keeps behaving sensibly before it tips into nonsense. Unlike a test that asks does the policy pass this situation, sensitivity analysis asks what is the policy's behaviour a function of — attributing the output's variation to its inputs, and in doing so revealing both how much to trust the model and whether its more elaborate parts earn their place or merely add detail that changes nothing.

Example¶

A fire agency uses a wildfire-spread model to decide when to order an evacuation — the model turns wind, fuel moisture, slope, and humidity into a predicted time-to-reach for each community. Before betting lives on it, analysts run a sensitivity sweep: each input is varied across its plausible range, alone and in combination, and they watch how the evacuate-now-or-hold decision moves.

Two findings change how the model is used. First, the decision is overwhelmingly driven by wind speed and fuel moisture, while three carefully-modelled vegetation sub-parameters barely move it at all — detail that added complexity without changing decisions, and that can be simplified or held fixed. Second, the model stays well-behaved across normal conditions but its predictions become wildly unstable once modelled wind exceeds ≈50 km/h — a validity boundary past which its output should not be trusted to time an evacuation, and where humans must take over. Neither finding is about any single scenario; both are about the shape of the model's dependence on its inputs.

How it works¶

Sensitivity analysis is defined by working on the model's input–output map, not on any single case:

Vary inputs, attribute the output. Parameters are swept across ranges — ideally many at once, not one at a time — and the variation in the decision is apportioned to the inputs that caused it.
Separate the load-bearing from the inert. The result ranks which few parameters dominate and which barely register, telling you where estimation effort belongs and where the model is over-built.
Find where behaviour breaks. The sweep exposes the range over which the model stays stable and the point past which it becomes erratic — a validity boundary, not a pass/fail on one scenario.
Prefer global to local. Sweeping the whole plausible space, rather than nudging one parameter around a base case, is what catches interactions where two inputs matter only together.

Tuning parameters¶

Sweep scope — one-at-a-time versus global, with all parameters varied together. Global sweeps catch interactions but cost far more runs; one-at-a-time is cheap but blind to coupling.
Range width — how far each input is pushed. Wider ranges reveal the validity boundary but spend effort on regions that may never occur.
Resolution — how finely the space is sampled. Finer grids find sharp cliffs but multiply computation.
Output of interest — whether sensitivity is measured on a raw prediction or on the decision it drives. A parameter can swing the prediction yet never flip the decision — measuring the decision is usually what matters.
Interaction depth — whether pairwise (or higher) interactions are estimated, not just main effects. Deeper interaction analysis finds hidden couplings but needs many more samples.

When it helps, and when it misleads¶

Its strength is that it tells you which inputs to measure well, which model detail to cut, and where the model's trustworthy range ends — cheaply, before any of those blind spots costs a real decision. It is the test that most directly answers "is this added modelling layer doing anything?"

The classic error is a one-at-a-time sweep that nudges each parameter around a single base case and declares the model robust — while missing interactions where two parameters are dangerous only in combination, exactly the regime a global, variance-based analysis is built to catch.^[1] Results are also only as honest as the ranges assumed: too-narrow ranges manufacture a reassuring robustness, and the method is easily run backwards to show a favoured model is "insensitive" by sweeping only where it is flat. The discipline is to sweep globally over honestly-wide ranges, report sensitivity on the decision and not just the prediction, and treat "insensitive" as a claim to stress-test rather than a comfort to accept.

How it implements the components¶

Sensitivity Analysis fills the robustness-and-validity slice — the components that describe how much to trust the model and where:

uncertainty_model — by attributing output variation to inputs, it characterizes how input uncertainty propagates to the decision and which parameters dominate that uncertainty.
validity_boundary — the sweep reveals the range over which the model stays well-behaved and the point past which its output can no longer be trusted.

It does not check named safety cases (Scenario Testing), grade the policy on real history (Historical Replay), or build the model whose parameters it perturbs (State-Space Model).

Instantiates: Model-Based Regulation — supplies the map of what the model's decisions actually depend on and where they stop being trustworthy.
Consumes: State-Space Model or Causal Loop Diagram supplies the model whose parameters it sweeps.
Sibling mechanisms: Scenario Testing · Historical Replay · Digital Twin Trial · Model-Failure Red Team · Forecast Backtesting

Notes¶

Robust is not the same as right. Sensitivity analysis can certify that a model's decisions barely move as its parameters vary — and that model can still be confidently, stably wrong about the real system. Insensitivity to parameters says the conclusion is well-determined by the model; it says nothing about whether the model corresponds to reality. Pair it with tests that confront the model with the world (Historical Replay, Scenario Testing) before reading low sensitivity as reassurance.

References¶

[1] Global (variance-based) sensitivity analysis attributes the variance of a model's output across the whole input space, capturing interactions that a one-at-a-time sweep around a fixed base case cannot see; Sobol indices are a standard formulation. The distinction matters because a model can look robust under one-at-a-time perturbation yet be fragile to two inputs moving together. ↩