Heartbeat and Suspicion Detector¶
Runtime failure detector — instantiates Assumption-Bounded Distributed Agreement
Continuously pings participants and maintains a per-node suspicion level, turning the raw stream of present-and-absent signals into the graded, revisable failure judgment that leader election and reconfiguration consume.
Heartbeat and Suspicion Detector is the running eye of the system: participants exchange periodic heartbeats, and from their arrival, absence, and timing it maintains a suspicion estimate for every peer. Its defining move is to separate observation from judgment — the heartbeats are raw signal, the suspicion is an interpretation — and, knowing that perfect failure detection is impossible in an asynchronous network, to emit a graded, revisable suspicion (often a continuous value) rather than a brittle dead-or-alive verdict. That gradation is what lets the rest of the system trade false alarms against detection speed by choosing where to act. This detector is the failure-detector abstraction made concrete, and the same heartbeat history doubles as the liveness audit trace operators read after the fact.
Example¶
A streaming-message cluster assigns each data partition to one broker. Brokers gossip heartbeats every ≈250 ms, and each maintains an accrual suspicion value for the others. One broker slips behind a brief network blip and misses two heartbeats. A naive binary detector would immediately declare it dead and trigger a costly partition reassignment — a self-inflicted outage. Instead the accrual detector merely raises its suspicion smoothly; when heartbeats resume within ≈1 second, suspicion decays back to zero and nothing happens. Later, a broker genuinely crashes: heartbeats stop for good, suspicion climbs past the action threshold, and the detector emits a "suspected" signal that the leader-election mechanism consumes to reassign that broker's partitions to a healthy peer. The graded signal absorbs the transient and reacts to the real failure — the difference between a flapping cluster and a stable one.[n1]
How it works¶
- Heartbeat, directly or indirectly. Participants ping on an interval; some detectors add indirect probing (ask a third node to ping a silent peer) so a single bad link does not masquerade as a dead node.
- Accrue, don't flip. An accrual function maps recent heartbeat history to a rising/falling suspicion value, rather than snapping between alive and dead.
- Threshold the action, not the belief. A separate, tunable action threshold decides when suspicion is high enough to report — keeping the underlying estimate continuous and revisable.
- Keep the trace. The heartbeat and suspicion history persists as the liveness record that monitoring and post-mortems consult.
Tuning parameters¶
- Heartbeat interval — faster beats detect failures sooner but add network traffic and raise the false-positive rate under jitter.
- Suspicion / action threshold — the level at which a peer is reported. Low thresholds react fast but jumpily; high thresholds are stable but slow.
- Direct vs. indirect probing — whether a silent peer is cross-checked through others before being suspected; indirect probing filters transient single-link failures at some latency cost.
- History window — how much past timing the accrual estimate integrates; longer windows are steadier, shorter ones more responsive.
- Recovery decay — how quickly suspicion falls when heartbeats resume, governing how readily a briefly-silent peer is forgiven.
When it helps, and when it misleads¶
Its strength is handing the rest of the system a tunable, continuous failure signal instead of a raw, brittle timeout — the knob that lets operators dial the false-alarm/detection-latency trade to fit the workload. Indirect probing further filters the transient link failures that would otherwise trigger needless failovers.
Its danger is that in a truly asynchronous network no detector can be both complete and accurate[1] — some real failures are missed and some healthy nodes are wrongly suspected. A partition makes a healthy remote node look identical to a dead one; correlated network events can trigger a suspicion storm and mass, simultaneous false failover. The classic misuse is driving irreversible actions straight off suspicion — fencing hardware, deleting data, reassigning ownership — as if suspicion were fact. The discipline is to treat suspicion as revisable evidence: require a safe or fenced handoff before acting on it, and back any irreversible step with a real quorum decision rather than the detector's word alone.
How it implements the components¶
failure_detector— it is the failure detector: heartbeats in, graded suspicion out, with completeness and accuracy tunable rather than assumed perfect.observability_and_audit_trace— its continuous heartbeat and suspicion history is the liveness trace operators and post-mortems read to reconstruct who was reachable when.
It does not set the wait durations it consumes (that is Timeout Policy), elect or rotate the leader its signal triggers (Term/Epoch Leader Election), or decide any committed value (Quorum or Consensus Commit).
Related¶
- Instantiates: Assumption-Bounded Distributed Agreement — it supplies the runtime failure signal that liveness-driven mechanisms react to.
- Consumes: Timeout Policy supplies the deadlines that turn missed heartbeats into rising suspicion.
- Sibling mechanisms: Timeout Policy · Term/Epoch Leader Election · Quorum or Consensus Commit · Byzantine Fault-Tolerant Quorum Protocol · Consensus Fault-Injection Test · Joint-Consensus Membership Change · Paxos-Style Quorum Protocol · Raft-Style Replicated-Log Protocol · Randomized Common-Coin Protocol · Signed Quorum Certificate · View-Change Protocol · Write-Ahead Vote Log
Editorial Notes¶
Form Classification¶
Form family: Monitoring, Sensing & Alerting
Rationale: Heartbeat and Suspicion Detector operates as an ongoing sensing arrangement that repeatedly observes actual state and surfaces changes or alerts because it continuously pings participants and maintains a per-node suspicion level, turning the raw stream of present-and-absent signals into the graded, revisable failure judgment that leader election and reconfiguration consume
Independent corroboration: The frozen evidence defines Heartbeat and Suspicion Detector as 'Continuously pings participants and maintains a per-node suspicion level, turning the raw stream of present-and-absent signals into the graded, revisable failure judgment that leader election and reconfiguration consume', so its operative form is Monitoring, Sensing & Alerting.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Computer Science & Software Engineering
Origin pattern: Single lineage
Present-day reach: Specialized
Rationale: Accrual suspicion from heartbeat timing is a named distributed-systems failure-detector lineage associated with Chandra-Toueg and phi-accrual detectors.
Review outcome: Independent reviewer agreement; high confidence.
Notes¶
Suspicion is not truth. The detector's output is deliberately fallible — a suspected node may be perfectly healthy on the far side of a partition — so system safety must never rest on suspicion alone. Any action gated on it should be safe even when the suspicion is wrong.
[n1] The φ-accrual failure detector (Hayashibara and colleagues) outputs a continuous suspicion level φ instead of a boolean, letting applications pick their own action threshold. It sits in the tradition of Chandra and Toueg's unreliable failure detectors, where even the weakest useful class (◊S) is only eventually accurate — a formal statement of why perfect detection is unattainable in asynchronous systems. ↩
References¶
[1] Chandra, T. D., & Toueg, S. "Unreliable Failure Detectors for Reliable Distributed Systems". Journal of the ACM 43(2), 225–267 (1996). Establishes the impossibility of a perfect failure detector in a purely asynchronous crash-failure model and the possibility of falsely suspecting a healthy process. registry ↩ Show verification details
SupportedVerified against the work's full text
Chandra and Toueg state that in completely asynchronous systems accurate failure detection is impossible, since crashed and very slow processes cannot be reliably told apart.
“it is impossible to reliably distinguish a process that has crashed from one that is merely very slow. In other words, Consensus is unsolvable because accurate failure detection is impossible.”