Failure Injection¶
Fault-injection technique — instantiates Chaos Exposure Testing
The actuator that delivers a specific, bounded fault into a component on demand — disabling, delaying, corrupting, or degrading it — with a kill-switch to stop and a defined path to undo.
Failure injection is the hands-on act of making the fault happen. It reaches into a component and disables it, delays it, corrupts its inputs, or degrades its output — deliberately, on command, to a named target. Its defining move is that it is a reversible actuator, not an experiment: it owns the mechanics of delivery (what fault, where, how to arm it), the stop control that yanks it back the instant the exposure crosses a line, and the restore path that returns the component to health. It answers how do we safely make this break? — not what should we conclude? That is why it can serve many masters: a Chaos Engineering Experiment borrows it to test a hypothesis, but so does a certification test-bench that simply needs to prove a safety mechanism trips. The injector supplies the fault and the undo; whatever frames the result sits above it.
Example¶
An automotive supplier is validating the anti-lock braking controller for a new car, and functional-safety standards require them to prove the controller detects and handles a stuck wheel-speed sensor. Reproducing that on a real vehicle would be dangerous, so the fault is injected on a hardware-in-the-loop rig: the ECU runs its real firmware while a simulator feeds it synthetic sensor signals. The injection is precise — at a scripted moment, the rig freezes the front-left wheel-speed value while the others keep changing, exactly the "stuck sensor" fault. Two things are wired in before the run: a stop condition (if the rig's own monitors see the test bench itself misbehave, everything halts) and a restore path (the injected signal is released and the ECU returns to nominal input). The controller flags the fault and falls back to conventional braking within the required window — proven, on a bench, without a car ever leaving the lab.[1]
How it works¶
- Specify the fault precisely. Name the target component, the fault mode (kill, latency, corruption, resource exhaustion), the timing, and the sequence — a vague "break something" is not injectable.
- Arm a stop control first. Wire the kill-switch before the fault, so the exposure can be aborted on a defined signal (safety limit breached, a human calling it) faster than damage propagates.
- Deliver, hold, release. Apply the fault, hold it long enough to exercise the response, then execute the restore path that returns the component to acceptable operation.
- Keep delivery separate from judgment. The injector reports only what it did and undid; interpreting the response belongs to whatever experiment or drill invoked it.
Tuning parameters¶
- Fault mode — which failure to simulate (crash, slow, corrupt, exhaust); each exercises a different recovery path, and picking the wrong mode tests a defense the real world never triggers.
- Injection point — how deep the fault sits — network edge, process, syscall, hardware signal; deeper points are more faithful but harder to bound and undo.
- Duration and intensity — how long and how hard the fault bites; longer and harder reveals slow-building failures but raises the odds of exceeding the stop condition.
- Reversibility guarantee — whether the undo is instantaneous, staged, or best-effort; a weaker restore path forces a smaller, more cautious injection.
- Targeting precision — one named instance versus a whole class of components; precise targeting keeps blast radius legible, broad targeting finds coupling but blurs cause.
When it helps, and when it misleads¶
Its strength is control: it can reproduce a specific, dangerous failure on demand, in a place where the consequences are bounded and reversible, which makes otherwise-untestable recovery paths testable. Because it is decoupled from any single framing, the same injector serves experiments, certification benches, and drills alike.
Its defining risk is that the stop and restore are promises — a "reversible" injection whose undo silently fails, or whose stop control is slower than the failure it triggers, turns the test into the incident. It also invites a subtler misuse: injecting only faults the system is already known to survive, to generate a green certification report — an injector arranged to pass. And an injector with no experiment or drill around it produces motion without meaning: a component broken and restored, with nothing concluded. The discipline that guards against this is to test the stop and rollback paths before trusting them with a real fault, to vary the fault modes rather than replay the comfortable one, and to never run an injection that is not attached to a question.
How it implements the components¶
perturbation_plan— it is the executable specification and delivery of the disturbance: what fault, where, when, in what sequence.guardrail_and_stop_condition— the armed kill-switch and abort trigger that halt the exposure the moment it crosses its defined boundary.rollback_policy— the restore path that removes the fault and returns the component to acceptable operation, which is what makes realistic injection safe to attempt.
It delivers and undoes the fault but never decides what the fault means: the falsifiable claim (fragility_hypothesis) and the numeric control it is read against (steady_state_baseline) belong to its nearest twin Chaos Engineering Experiment, which reasons where this injector acts.
Related¶
- Instantiates: Chaos Exposure Testing — it is the reversible actuator that makes a bounded fault happen on command.
- Sibling mechanisms: Chaos Engineering Experiment · Observability Dashboard · Fire Drill · Red-Team Stress Test · Runbook Rehearsal · Disaster Exercise · Canary Perturbation
Editorial Notes¶
Form Classification¶
Form family: Experiment, Test & Rehearsal
Rationale: The injector deliberately applies a precisely bounded fault to generate evidence about system response, with an armed stop control and defined restoration path.
Nearest alternative: Intervention, Treatment & Transformation — It directly changes a component temporarily, but the purpose is controlled exposure for learning rather than lasting target improvement.
Review outcome: Adjudicated after independent review; high confidence.
Origin Attribution¶
Primary origin: Computer Science & Software Engineering
Origin pattern: Convergent development
Present-day reach: Specialized
Rationale: Programmatically injecting delay, corruption, outage, or degradation is canonical software fault-injection and chaos-engineering practice.
Related originating lineages:
- Engineering & Design — Reliability engineering independently developed deliberate component fault insertion in physical and electronic systems. Hardware fault-injection and reliability testing materially predate and shape bounded fault delivery and recovery controls.
Review resolution: Both reviewers agree that computer_science is primary. I retain engineering_design only as formative origin lineages; convergent is appropriate because the same operational pattern arose through parallel professional lineages. Reach is specialized because the mechanism remains tied to a bounded professional technique, an applicability judgment kept separate from provenance. Encyclopedia synthesis is false because the artifact is already established enough that encyclopedia-specific synthesis is not required. No unresolved historical ambiguity remains after reconciling the secondary fields.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
Failure injection is an actuator, not a verdict: on its own it proves nothing, because breaking and restoring a component says nothing about resilience until a hypothesis or a drill gives the response a meaning. Keeping the injector distinct from the framing above it is what lets the same tool serve a research experiment one day and a compliance bench the next without changing what it is.
References¶
[1] Fault-injection testing — deliberately introducing faults to verify that error-detection and recovery mechanisms work — is a long-standing dependability-engineering technique, required by safety standards such as ISO 26262 for road-vehicle electronics. Hardware-in-the-loop rigs are its standard safe venue for hazardous faults. registry ↩