Timeout Policy¶

Waiting-and-stall policy — instantiates Assumption-Bounded Distributed Agreement

Bounds how long a participant will wait for an expected message, and converts the resulting silence into a safe action — abort, retry, step down, stall — never into a claim about who has failed.

Timeout Policy is the mechanism that gives an asynchronous system its only handle on the passage of time: a maximum duration any participant will wait for an expected message — a vote, an acknowledgement, a heartbeat — before it must act on the absence of that message. Its defining subtlety, and the thing that separates a disciplined timeout from a naive one, is that a fired timeout is fundamentally ambiguous: it cannot distinguish a participant that has crashed from one that is merely slow or sitting behind a partition. So the policy's job is not to decide "that node is dead." It is to pick, in advance, an action that stays safe whichever is true — abort the transaction, retry, step down as leader, or declare a visible stall — so the system never converts silence into an unsafe guess.

Example¶

An order-processing transaction spans three services — charge payment, reserve inventory, book shipping — coordinated by a two-phase commit. The coordinator sends prepare and waits for each service's vote. Payment and shipping vote yes; inventory says nothing. The coordinator's timeout fires at, say, ≈2 seconds. It has no way to tell whether inventory has crashed or is just paused behind a long garbage-collection stall or a brief network hiccup — and committing without inventory's yes could oversell the last unit in stock. So the policy's pre-chosen safe action on expiry is abort: release the payment hold, cancel the tentative reservation, and let the order be retried cleanly. Crucially, the coordinator does not declare inventory permanently dead and reroute around it; it simply refuses to commit under uncertainty and gives up the round safely. Whether inventory was dead or merely slow, no partial, unsafe commit ever happened.

How it works¶

Turn the delay model into a number. The assumed message-delay and partition behaviour is compiled into concrete wait bounds — one clock the protocol can actually consult.
Treat expiry as a trigger, not a verdict. When the timer fires, the policy does not assert failure; it executes the branch that is correct whether or not the peer is alive.
Default toward safety. Where the safe action and the fast action diverge, the policy favours the safe one (abort, hold, step down) — accepting a slower recovery to avoid an unsafe commit or a needless leadership change.

It owns the clock and the reaction to silence. It does not judge which specific participant has failed, nor how confident that judgment is — that is a separate role.

Tuning parameters¶

Duration — short timeouts react fast but false-suspect healthy-but-slow peers; long ones are calm but leave the system hanging. This is the central trade and it is rarely one-size-fits-all.
Fixed vs. adaptive — a constant bound, or one that tracks observed round-trip time so it widens under load. Adaptive timeouts cut false positives but can be gamed by a slowly-degrading network.
Backoff and jitter — how retries space out after an expiry, to avoid a synchronized retry storm hammering a recovering peer.
Per-message-class bounds — separate timeouts for votes, heartbeats, and catch-up transfers, since each has a different natural latency.
The safe-default branch — which action expiry triggers (abort vs. hold vs. step down). This encodes the system's safety-over-liveness preference.

When it helps, and when it misleads¶

Its strength is that it is the cheap, universal lever that keeps a distributed system live: without a bound on waiting, one silent participant can freeze everything indefinitely. It forces every wait to end in a bounded, deliberate reaction.

Its danger is treating the timeout as evidence of failure, which it is not — no asynchronous protocol can reliably tell a crashed node from a slow one.^[1] Set too aggressively, timeouts flood the system with false suspicions, causing needless aborts and leadership churn — a leader that pauses for a long GC gets "failed over" and the cluster flaps. Set too slackly, the system stalls whenever anything goes quiet. The classic misuse is tuning timeouts down to hit a latency target until the system spends its life false-suspecting healthy nodes. The discipline is to keep expiry meaning "act safely under uncertainty," to pair the clock with a real failure detector for the judgment of who failed, and to size the bound against the assumed message-delay budget rather than the SLA you wish you had.

How it implements the components¶

failure_timeout_and_partition_model — it turns the assumed delay/partition behaviour into the concrete wait bounds the protocol runs on; the policy is that model made operational.
stall_and_partition_policy — it fixes what happens when a timeout fires during a suspected partition: a visible stall or safe abort in preference to unsafe progress.

It does not judge which participant has actually failed or hold a per-node suspicion level (that is Heartbeat and Suspicion Detector), nor bring a recovered participant back into the group (recovery is handled by View-Change Protocol and Write-Ahead Vote Log).

Instantiates: Assumption-Bounded Distributed Agreement — it supplies the bounded-waiting behaviour every other mechanism relies on to make progress at all.
Sibling mechanisms: Heartbeat and Suspicion Detector · Quorum or Consensus Commit · Byzantine Fault-Tolerant Quorum Protocol · Consensus Fault-Injection Test · Joint-Consensus Membership Change · Paxos-Style Quorum Protocol · Raft-Style Replicated-Log Protocol · Randomized Common-Coin Protocol · Signed Quorum Certificate · Term/Epoch Leader Election · View-Change Protocol · Write-Ahead Vote Log

Notes¶

Timeout and failure detection are constantly conflated but are two different roles. The timeout is the clock — when to stop waiting. The Heartbeat and Suspicion Detector is the judgment — who to suspect, and how strongly. Keeping them separate lets you tune how fast the system reacts to silence without changing what it believes about failure, and vice versa.

References¶

[1] The FLP impossibility result (Fischer, Lynch, and Paterson) shows that in a fully asynchronous system with even one crash fault, no deterministic protocol can guarantee agreement, precisely because a slow participant is indistinguishable from a failed one. Timeouts are the practical heuristic used to sidestep this — which is exactly why their output must be treated as a trigger for safe action, not as truth. ↩