Heartbeat and Suspicion Detector¶

Runtime failure detector — instantiates Assumption-Bounded Distributed Agreement

Continuously pings participants and maintains a per-node suspicion level, turning the raw stream of present-and-absent signals into the graded, revisable failure judgment that leader election and reconfiguration consume.

Heartbeat and Suspicion Detector is the running eye of the system: participants exchange periodic heartbeats, and from their arrival, absence, and timing it maintains a suspicion estimate for every peer. Its defining move is to separate observation from judgment — the heartbeats are raw signal, the suspicion is an interpretation — and, knowing that perfect failure detection is impossible in an asynchronous network, to emit a graded, revisable suspicion (often a continuous value) rather than a brittle dead-or-alive verdict. That gradation is what lets the rest of the system trade false alarms against detection speed by choosing where to act. This detector is the failure-detector abstraction made concrete, and the same heartbeat history doubles as the liveness audit trace operators read after the fact.

Example¶

A streaming-message cluster assigns each data partition to one broker. Brokers gossip heartbeats every ≈250 ms, and each maintains an accrual suspicion value for the others. One broker slips behind a brief network blip and misses two heartbeats. A naive binary detector would immediately declare it dead and trigger a costly partition reassignment — a self-inflicted outage. Instead the accrual detector merely raises its suspicion smoothly; when heartbeats resume within ≈1 second, suspicion decays back to zero and nothing happens. Later, a broker genuinely crashes: heartbeats stop for good, suspicion climbs past the action threshold, and the detector emits a "suspected" signal that the leader-election mechanism consumes to reassign that broker's partitions to a healthy peer. The graded signal absorbs the transient and reacts to the real failure — the difference between a flapping cluster and a stable one.^[1]

How it works¶

Heartbeat, directly or indirectly. Participants ping on an interval; some detectors add indirect probing (ask a third node to ping a silent peer) so a single bad link does not masquerade as a dead node.
Accrue, don't flip. An accrual function maps recent heartbeat history to a rising/falling suspicion value, rather than snapping between alive and dead.
Threshold the action, not the belief. A separate, tunable action threshold decides when suspicion is high enough to report — keeping the underlying estimate continuous and revisable.
Keep the trace. The heartbeat and suspicion history persists as the liveness record that monitoring and post-mortems consult.

Tuning parameters¶

Heartbeat interval — faster beats detect failures sooner but add network traffic and raise the false-positive rate under jitter.
Suspicion / action threshold — the level at which a peer is reported. Low thresholds react fast but jumpily; high thresholds are stable but slow.
Direct vs. indirect probing — whether a silent peer is cross-checked through others before being suspected; indirect probing filters transient single-link failures at some latency cost.
History window — how much past timing the accrual estimate integrates; longer windows are steadier, shorter ones more responsive.
Recovery decay — how quickly suspicion falls when heartbeats resume, governing how readily a briefly-silent peer is forgiven.

When it helps, and when it misleads¶

Its strength is handing the rest of the system a tunable, continuous failure signal instead of a raw, brittle timeout — the knob that lets operators dial the false-alarm/detection-latency trade to fit the workload. Indirect probing further filters the transient link failures that would otherwise trigger needless failovers.

Its danger is that in a truly asynchronous network no detector can be both complete and accurate — some real failures are missed and some healthy nodes are wrongly suspected.^[1] A partition makes a healthy remote node look identical to a dead one; correlated network events can trigger a suspicion storm and mass, simultaneous false failover. The classic misuse is driving irreversible actions straight off suspicion — fencing hardware, deleting data, reassigning ownership — as if suspicion were fact. The discipline is to treat suspicion as revisable evidence: require a safe or fenced handoff before acting on it, and back any irreversible step with a real quorum decision rather than the detector's word alone.

How it implements the components¶

failure_detector — it is the failure detector: heartbeats in, graded suspicion out, with completeness and accuracy tunable rather than assumed perfect.
observability_and_audit_trace — its continuous heartbeat and suspicion history is the liveness trace operators and post-mortems read to reconstruct who was reachable when.

It does not set the wait durations it consumes (that is Timeout Policy), elect or rotate the leader its signal triggers (Term/Epoch Leader Election), or decide any committed value (Quorum or Consensus Commit).

Instantiates: Assumption-Bounded Distributed Agreement — it supplies the runtime failure signal that liveness-driven mechanisms react to.
Consumes: Timeout Policy supplies the deadlines that turn missed heartbeats into rising suspicion.
Sibling mechanisms: Timeout Policy · Term/Epoch Leader Election · Quorum or Consensus Commit · Byzantine Fault-Tolerant Quorum Protocol · Consensus Fault-Injection Test · Joint-Consensus Membership Change · Paxos-Style Quorum Protocol · Raft-Style Replicated-Log Protocol · Randomized Common-Coin Protocol · Signed Quorum Certificate · View-Change Protocol · Write-Ahead Vote Log

Notes¶

Suspicion is not truth. The detector's output is deliberately fallible — a suspected node may be perfectly healthy on the far side of a partition — so system safety must never rest on suspicion alone. Any action gated on it should be safe even when the suspicion is wrong.

References¶

[1] The φ-accrual failure detector (Hayashibara and colleagues) outputs a continuous suspicion level φ instead of a boolean, letting applications pick their own action threshold. It sits in the tradition of Chandra and Toueg's unreliable failure detectors, where even the weakest useful class (◊S) is only eventually accurate — a formal statement of why perfect detection is unattainable in asynchronous systems. ↩