Skip to content

Partition and Clock Fault Injection

Fault-injection test — instantiates Shared-State Consistency Contract Design

Deliberately induces network partitions, message delays, and clock skew against a running system to test whether its consistency contract actually holds under the faults it claims to tolerate.

Partition and Clock Fault Injection manufactures the adverse conditions a consistency contract quietly assumes away — network partitions, asymmetric links, message delay, clock skew and jumps, process pauses — while a real workload runs, then checks whether the promised guarantee survives. Its defining role is to exercise and falsify assumptions rather than to define, measure, or enforce a guarantee. Contracts routinely rest on implicit clauses ("clocks stay within X," "partitions heal quickly," "leases are safe"); this mechanism turns each such clause into a test, drives the system past it, and watches for a violation — converting hand-waved fault tolerance into evidence, and keeping the assumption register honest by writing back what actually held.

Example

A store uses lease-based leader election and assumes clock skew stays under 500 ms — a comfortable clause nobody has tested. The fault injector freezes the leader's process for three seconds (a GC-pause analog), jumps its clock forward, and simultaneously partitions it from the majority. Now two nodes can briefly believe they hold a valid lease: the paused old leader wakes thinking its lease is still good while a new leader has already been elected. Paired with a linearizability oracle, the run catches a window where both accept writes — a split-brain violation the happy path never exposes. The finding is concrete: the 500 ms skew assumption is unsafe without fencing tokens, and that falsified clause, plus its fix, is recorded as an entry in the assumption register. Tools such as Jepsen exist precisely to run this kind of test.[1]

How it works

The distinguishing move is deriving faults from the declared assumptions and pushing past them. It enumerates the assumptions to attack (from the register), schedules fault scenarios — partition topologies, clock manipulations, delays, pauses — around a running workload, and pairs each run with a violation oracle that decides whether the contract broke. When a violation is found, it minimizes the schedule to the smallest reproducer. It targets the two assumptions that silently kill consistency most often: clock behavior and partition behavior. It does not itself define the model being checked — it supplies the faults; the oracle supplies the verdict.

Tuning parameters

  • Fault menu — clean partition, asymmetric partition, flaky link, clock skew/jump, process pause: realism versus blast radius.
  • Fault magnitude & timing — how large the skew, how long the partition, and where injected: how hard the assumptions are pushed.
  • Workload contention — concurrency during injection: higher contention surfaces rarer interleavings.
  • Oracle strictness — which consistency model the run is checked against: sensitivity of the test.
  • Environment — staging vs. production game-day: realism versus operational safety.

When it helps, and when it misleads

Its strength is falsifying over-optimistic assumptions before customers do. Partition and clock behavior are nearly impossible to exercise with ordinary tests, so this is often the only way to learn whether "we tolerate partitions" is true, and it turns a vague claim into a reproducible failure.

It misleads when a clean run is read as a proof. Fault injection only explores the schedules it actually injects, so "no violation found" is not "no violation exists"; a weak oracle passes real bugs (garbage-in); and the whole exercise can be run backwards — injecting only mild, survivable faults so the system "passes" and the assumption gets a rubber stamp. Careless injection in production is itself a hazard. The discipline is to derive faults from the declared assumptions and push beyond them, pair with a strong oracle, and treat green as "not yet falsified," never as verified.

How it implements the components

  • fault_partition_and_clock_assumptions — makes the contract's fault, partition, and clock assumptions executable and drives the system past each one to test it.
  • assumption_register — operationalizes the register: every clause becomes a test and every falsified assumption is written back as an entry, keeping the list of "what we're relying on" honest.

It does not define the consistency model it checks against (legal_history_contract, consistency_violation_oracle) — those are Sequential-Consistency Trace Protocol and Consistency History Checker — nor implement the guarantee under test (Linearizable Read/Write Protocol) or the topology it partitions (Leader-Based Replication).

References

[1] Jepsen (Kyle Kingsbury) is the best-known consistency fault-injection tool: it applies partitions and clock skew while checking recorded histories against a consistency model, and has publicly falsified strong-consistency claims of numerous production databases. ↩