Skip to content

Consensus Fault-Injection Test

Fault-injection test / resilience audit — instantiates Assumption-Bounded Distributed Agreement

Deliberately injects the faults a consensus protocol claims to tolerate — crashes, delays, partitions, reordering — to check that agreement stays safe inside its assumption budget and degrades to a visible stall outside it.

Every protocol in this archetype rests on a stated assumption budget — how many faults, what kind, what timing. A Consensus Fault-Injection Test is the mechanism that refuses to take that budget on faith and attacks it. Rather than testing that the system works when things go right, it manufactures exactly the adversity the protocol claims to survive — killing participants mid-round, pausing processes, partitioning the network, delaying and reordering messages, skewing clocks — and asserts the safety invariant throughout: that no two conflicting decisions are ever observed. Its defining discipline is the split it insists on between two outcomes. A safety violation (two committed values that disagree) must never occur, inside the budget or out. A liveness stall (the system going visibly undecided) is acceptable — indeed correct — once faults exceed the liveness assumptions. The test is what turns an assumption register from a document into a set of claims that have actually been shot at.

Example

A site-reliability team is hardening a replicated datastore before it carries production traffic. In staging they run a fault-injection campaign: a scripted "nemesis" isolates the leader into a minority partition, pauses it mid-commit with a stop signal, reorders acknowledgements, and heals the partition at unlucky moments — while a separate workload reads and writes continuously. Throughout, the harness asserts linearizability: no client may ever see two different committed values for the same key. Most runs end in the correct way — during the partition the minority side simply stalls, unable to reach a quorum, then catches up when the network heals. But one run surfaces a genuine bug: after a partition heals, a lagging replica briefly serves a stale committed value — a real safety violation. The team fixes the recovery path before shipping. This adversarial style of consensus testing is what tools in the Jepsen tradition were built to do.[1]

How it works

  • Derive the fault schedule from the budget. The faults injected are read straight off the protocol's own declared assumptions — inject everything inside the budget, plus a few probes just outside it to confirm the system fails safe rather than lying.
  • Assert continuously, not just at the end. A safety invariant (linearizability, or a weaker agreed property) is checked across the whole run, because violations often appear only during a partition or a heal, then vanish.
  • Separate the two verdicts. Every observed anomaly is classified: a safety break is a hard failure; a liveness stall under injected partition is expected behaviour, not a bug.
  • Feed the retrospective. Counterexamples, traces, and the exact schedule that produced them are handed to the post-agreement review to decide what must change.

Tuning parameters

  • Fault palette — which faults and how severe (clean crashes only, or the full menu of pauses, partitions, clock skew, and message corruption). A richer palette finds more, at more setup cost.
  • Schedule strategy — random/chaos, targeted-adversarial, or exhaustive/model-checked interleavings. Randomness is cheap and broad; targeted and exhaustive search are dearer but catch rare orderings.
  • Inside- vs. beyond-budget mix — how much effort probes the tolerated region versus the region past it (where only safe degradation is required).
  • Invariant strength — linearizability versus a weaker consistency check. Stronger invariants catch more but demand a correct oracle to check against.
  • Coverage vs. cost — run duration and the number of distinct interleavings explored, traded against time and infrastructure.

When it helps, and when it misleads

Its strength is that it surfaces the failures that only appear under partition and adverse timing — precisely the ones ordinary tests, which run in the happy synchronous case, never reach. It converts "we believe it is safe" into evidence, and its beyond-budget probes confirm the system fails safe rather than silently wrong.

Its limits are the limits of all testing, sharpened here. A passing run proves no counterexample was found, not that none exists; coverage is sampled, so a rare interleaving can escape. It gives false comfort if the injected faults do not resemble the fault distribution production will actually see. The classic misuses are running it once to tick a box and quietly narrowing the fault palette until the suite goes green. The discipline is to derive faults from the real assumption budget, always keep the beyond-budget probes, and read a green run as "no counterexample yet" — never as proof of safety.

How it implements the components

  • assumption_register — it is the register's adversary and its proving ground: it enumerates each declared assumption and attacks it, converting the register into claims that have been tested or falsified.
  • post_agreement_review — its counterexamples and traces feed the retrospective that decides whether the assumption budget, the protocol, or the deployment must change.

It does not declare the fault model or budget in the first place (the protocol under test does, e.g. Byzantine Fault-Tolerant Quorum Protocol), detect failures in live operation (Heartbeat and Suspicion Detector), or perform the commit it stresses (Quorum or Consensus Commit).

Notes

The safety-versus-liveness asymmetry is the whole point and the easiest thing to get wrong. A safety violation is a hard failure the harness must never tolerate; a liveness stall under an injected partition is the correct response, not a defect. Teams that confuse the two "fix" safe stalls by adding unsafe shortcuts — turning a passing test into a broken system.

References

[1] Jepsen (Kyle Kingsbury) is a well-known approach and toolkit for testing distributed systems by inducing network partitions and other faults while checking consistency invariants; it has a track record of finding real consensus and isolation violations in shipped databases. It is cited here as a genuine methodology, not for any specific claimed result. ↩