Chaos Engineering Experiment¶
Experiment — instantiates Chaos Exposure Testing
Runs a hypothesis-driven experiment on a live distributed system — inject turbulence, compare the disturbed behavior against a measured steady state, and let the difference confirm or refute a specific fragility claim.
A chaos engineering experiment treats resilience the way a lab treats a drug: as a claim to be tested, not asserted. It begins with a steady-state definition — a measurable picture of the system behaving normally (request success rate, checkout latency, orders-per-minute) — and a falsifiable hypothesis: "when we degrade dependency X, steady state will hold." Then it injects a real disturbance into the running system and watches whether the hypothesis survives. Its defining move is the controlled comparison: the value is not in breaking something but in the gap between predicted and observed behavior, read against a baseline that says what "fine" looks like. Where its sibling Failure Injection is the tool that delivers the fault, a chaos experiment is the surrounding scientific frame — the hypothesis, the control, and the disciplined minimization of blast radius that turns a fault into evidence.
Example¶
A video-streaming platform believes its recommendation service degrades gracefully: if the personalization backend is slow, the app should fall back to a generic "popular now" row and nobody should notice. That belief has never been tested in production. The team writes it as a hypothesis — "with 300 ms of added latency on the personalization call, playback-start success stays within 0.5% of baseline" — and first measures the steady state on a normal evening: 99.4% of sessions start playback within two seconds. They scope the experiment to 1% of traffic in a single region, run it for ten minutes, and have the injector apply the 300 ms of latency. The dashboard shows playback-start holding steady, exactly as predicted — but a second metric they had the foresight to watch, thumbnail load, collapses, because the fallback row still calls personalization for images. The graceful-degradation claim was half true. The experiment refuted a hidden corner of it at the cost of ten minutes and one percent of one region.
How it works¶
- Define steady state as numbers, not adjectives. Pick the handful of metrics that mean "the system is serving users," and measure their normal range first — the experiment is meaningless without the control.
- State the hypothesis in the null form. Write down the outcome you expect the perturbation to produce ("steady state holds"), so an unexpected result is unambiguous rather than rationalizable after the fact.[n1]
- Minimize blast radius on purpose. Start at the smallest scope that can still move the metric — a single percent of traffic, one availability zone — because the point is evidence, not damage, and scope can always be widened after a clean run.
- Compare, then widen or stop. Read the disturbed metrics against the baseline. A surviving hypothesis earns a larger blast radius next time; a refuted one becomes a finding.
Tuning parameters¶
- Blast-radius fraction — what share of traffic, users, or nodes the experiment touches; larger scope exercises more real coupling but raises the cost of a wrong hypothesis.
- Hypothesis specificity — a narrow single-metric claim versus a broad "nothing bad happens"; narrow claims give crisp refutations, broad ones catch surprises but are harder to read.
- Steady-state metric set — how many and which signals define "normal"; too few and the experiment declares success while a side effect burns, too many and noise drowns the result.
- Automation and cadence — a one-off manual run versus a continuous scheduled experiment in the deployment pipeline; automation catches regressions but demands airtight guardrails to run unattended.
- Environment fidelity — production, a production-like staging twin, or a load-shadowed replica; closer to production is more honest and more dangerous.
When it helps, and when it misleads¶
Its strength is that it converts a comfortable belief ("we degrade gracefully") into a checkable result, and it does so at a scope small enough that being wrong is survivable. It is the only mechanism here whose output is evidence about a specific claim rather than a general rehearsal, which makes it uniquely good at killing overconfident assumptions before an incident does.
Its central failure mode is a badly chosen steady state: if the baseline metrics miss the thing that actually breaks — as the thumbnail regression nearly was — the experiment returns a false "resilient" and manufactures the very false confidence the archetype warns against. It also tempts teams to run only hypotheses they expect to pass, producing a wall of green experiments that prove nothing, and it cannot test coupling it does not think to perturb. The discipline that guards against this is to watch more metrics than the hypothesis names, to prefer hypotheses the team is genuinely unsure of, and to treat an experiment that never refutes anything as a sign the perturbations are too gentle rather than the system invincible.
How it implements the components¶
fragility_hypothesis— the experiment is a falsifiable fragility claim written before the run, which is what separates it from undirected breakage.steady_state_baseline— it measures normal behavior as an explicit numeric control, without which the disturbed reading cannot be interpreted.blast_radius_limit— the "minimize blast radius" discipline is built into the design: the smallest scope that still moves the metric.
It designs and reads the experiment but does not deliver the fault, arm the kill-switch, or restore service — perturbation_plan, guardrail_and_stop_condition, and rollback_policy belong to its nearest twin Failure Injection, which acts where this experiment reasons. The standing live reading surface (observability_instrumentation) is Observability Dashboard's.
Related¶
- Instantiates: Chaos Exposure Testing — it supplies the hypothesis-and-control discipline that makes a perturbation into evidence.
- Consumes: Failure Injection delivers the fault the experiment reasons about; Observability Dashboard renders the steady-state signals it reads.
- Sibling mechanisms: Failure Injection · Observability Dashboard · Red-Team Stress Test · Runbook Rehearsal · Disaster Exercise · Fire Drill · Canary Perturbation
Editorial Notes¶
Form Classification¶
Form family: Experiment, Test & Rehearsal
Rationale: Runs a hypothesis-driven experiment on a live distributed system — inject turbulence, compare the disturbed behavior against a measured steady state, and let the difference confirm or refute a specific fragility claim, making its operative form a bounded trial, probe, simulation, or adversarial exercise that generates evidence from performance.
Independent corroboration: The frozen evidence defines Chaos Engineering Experiment as 'Runs a hypothesis-driven experiment on a live distributed system — inject turbulence, compare the disturbed behavior against a measured steady state, and let the difference confirm or refute a specific fragility claim', so its operative form is Experiment, Test & Rehearsal.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Computer Science & Software Engineering
Origin pattern: Single lineage
Present-day reach: Specialized
Rationale: Distributed-systems operations cohered chaos engineering as hypothesis-driven fault injection against a measured steady state in running systems.
Related originating lineages:
- Statistics & Experimental Design — Experimental method supplies falsifiable hypotheses, controlled disturbance, and comparison to baseline.
Review resolution: Computer science is the agreed primary lineage because chaos engineering emerged as hypothesis-driven failure injection in live distributed systems. Experimental statistics contributes controlled comparison against steady state, while the mechanism remains specialized to computing systems.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
[n1] The Principles of Chaos Engineering frame each run around a "steady-state hypothesis" and the deliberate minimization of blast radius — the conventions this mechanism follows, popularized by Netflix's resilience-testing tooling. The point is disproving a resilience claim through experiment, not causing outages. ↩