Failure-Injection Test¶

Fault-injection test — instantiates Eventual-Occurrence Containment Design

Deliberately induces a fault in the real system to confirm that detection, isolation, and failover actually fire as designed — proving the defensive chain before a real event exercises it.

Every containment design carries an unstated assumption: that when the moment comes, the detectors will notice, the isolation will trip, and the failover will hold. Failure-Injection Test refuses to take that on faith. It induces a controlled fault in the real (or realistically faithful) system on purpose and watches whether the whole defensive chain actually activates as designed — did the alarm fire in time, did the boundary close, did the system fall to its safe state — and it records every place the response fell short of the design. Its distinguishing question is do the reflexes fire? That is a different dial from how far a failure would spread (the blast-radius sibling) and how likely the fault is (the analytic siblings). Failure-Injection Test is the empirical proving ground: it turns "we believe the containment works" into "we watched it work — and here is where it didn't."

Example¶

Before a spacecraft ships, its fault-management software has to be trusted to handle failures with no one able to intervene in real time. On a hardware-in-the-loop testbed, engineers inject a specific fault — an attitude sensor that freezes on its last reading — and watch the onboard FDIR (Fault Detection, Isolation, and Recovery) logic respond. The design says: detect the frozen sensor within a set window, isolate it, switch to the redundant unit, and — if reconfiguration fails — drop to a safe hold mode. The injection reveals that detection fires correctly for a dead sensor but not for a stuck-but-plausible one, whose frozen value stays inside the sanity range long enough to slip past the check. That gap — a residual failure mode the design missed — goes onto the record and is fixed before launch, closing a hole that no amount of diagram review had surfaced. What the test proved was not that the system is safe, but exactly where it wasn't.

How it works¶

The defining move is to cause the failure deliberately rather than wait for it, then measure the actual response against the designed response, fault by fault. Each injection is a small experiment: pick a fault, apply it under controlled and abortable conditions, observe whether detection, escalation, isolation, and safe-state entry happen as specified and within their time budgets, and log the delta. The deliverable is not a green checkmark but a record of gaps — the residual failure modes that slipped through live controls — which is the honest input to the next round of hardening. Its software cousin is chaos engineering^[1], which injects failures into running production systems on scheduled "game days" for the same reason.

Tuning parameters¶

Fault realism and severity — mild, known faults versus severe, novel ones. Mild injections produce reassuring green results and little assurance; severe and unexpected ones are where real gaps surface.
Injection environment — dedicated testbed versus live production. Production is the truest test of whether the reflexes fire under real conditions, but carries the risk of causing the very outage you fear.
Blast containment of the test — how tightly scoped and abortable each injection is. A well-bounded injection can be stopped the instant it misbehaves; an open-ended one risks turning a test into an incident.
Coverage — which fault modes to inject and how exhaustively. Broad coverage finds more but costs more; narrow coverage is fast but leaves untested modes.
Cadence — one-shot validation versus continuous, recurring game-days. Recurring injection catches regressions as the system changes; one-shot proves only the version you tested.

When it helps, and when it misleads¶

Its strength is that it converts belief into evidence and drags dormant gaps into daylight — the detector that never actually fires, the failover that hangs, the alert that pages a rotation nobody staffs. Its failure modes are all species of theater. The classic misuse is to scope the injection so narrowly it cannot fail — inject only the faults you already know are handled — producing a passing result run backwards to justify a launch or a sign-off already decided. Testing only known modes leaves the unknown ones exactly where they were, and a careless injection in production can cause a real outage of its own. The discipline is to inject adversarially chosen and genuinely novel faults, keep every injection scoped and abortable, and treat each unhandled injection as a required fix rather than a footnote.

How it implements the components¶

Failure-Injection Test realizes the proving slice of the archetype's detection-and-response machinery — the parts an active test can verify and expose:

occurrence_detection_and_escalation_signal — it directly exercises the detection-and-escalation path, confirming the signal actually fires, and fires in time, when the event is induced rather than merely assumed.
residual_prevention_risk_record — its central deliverable is the record of gaps: the residual failure modes that survived the live controls and now need closing.

It proves the reflexes but does not keep the standing watch (Sentinel Event Monitoring), enact the isolation it exercises (Automatic Isolation Trip), measure how far a real failure would spread (Blast-Radius Test), or prove the full restore path (Recovery Drill and Restore Test).

Instantiates: Eventual-Occurrence Containment Design — the test is how the archetype's "prove recovery" and detection requirements are made real rather than assumed.
Sibling mechanisms: Blast-Radius Test · Automatic Isolation Trip · Recovery Drill and Restore Test · Degraded-Mode Runbook · Sentinel Event Monitoring · Fault Tree with Repeated-Opportunity Branch · Cumulative Risk Horizon Table · Probabilistic Safety Assessment · Repeated-Trial Probability Calculator · Opportunity Exposure Register · Post-Incident Recurrence Review · Stop-or-Scale-Back Gate

Notes¶

Failure-Injection Test and Blast-Radius Test both begin by breaking something on purpose, and they are easy to conflate — but they read different dials. Injection asks did the defenses fire?; blast-radius asks how far did the damage reach? Run both: a system whose reflexes fire flawlessly can still have an unacceptable radius, and a system with a tight radius can still fail to notice the event at all.

References¶

[1] Chaos engineering — the practice, popularized by Netflix's "Chaos Monkey," of deliberately injecting failures into running production systems to verify that resilience mechanisms respond as intended — is the software-industry form of failure-injection testing. ↩