Skip to content

Timeout and Retry Recovery

Recovery protocol — instantiates Deadlock Resolution

Caps every wait with a clock: when a participant has waited too long it abandons its blocked attempt, drops back to a controlled state, and retries — dissolving deadlocks no one ever detected.

Not every deadlock is worth detecting; some are cheaper to simply time out of. Timeout and Retry Recovery puts a clock on every wait. If a participant stays blocked past a threshold, it gives up the current attempt, releases what it was holding, falls back to a controlled starting state, and tries again later. Its defining move is that it is time-triggered, not diagnosis-triggered: it never builds a wait-for graph or names a cycle — it assumes that a wait this long is probably a deadlock (or indistinguishable from one that matters) and breaks it by having the waiter unilaterally abandon. That makes it wonderfully decentralized and wonderfully prone to livelock, where everyone times out and retries in lockstep and re-collides.

Example

Two services coordinate through a distributed lock service to update a shared record. svc-A acquires lock 1 and blocks trying to acquire lock 2; svc-B holds lock 2 and blocks on lock 1. No global detector is watching — but each acquisition attempt carries a timeout. After ~200 ms svc-A's wait on lock 2 expires; it releases lock 1, returns to the clean pre-attempt state it saved, and schedules a retry. That release lets svc-B acquire lock 1, finish, and drop both. When svc-A retries, the path is clear.

The failure this design must engineer around is the mirror case: if svc-A and svc-B time out at the same instant, both release, both retry immediately, and both re-deadlock — forever making motions without progress. The fix is not detection but randomization: each retry waits a randomized, growing interval (exponential backoff with jitter), so the two attempts desynchronize and one wins. The clock breaks the deadlock; the jitter keeps the break from repeating.

How it works

Its distinguishing property is that it needs no knowledge of the cycle at all:

  • Bound every wait. Attach a timeout to each blocking acquisition or request. The threshold is the only "detector" — a wait past it is treated as a break-worthy stall.
  • Abandon and release on expiry. When the clock fires, the waiter drops its blocked attempt and releases the resources it was holding, unilaterally cutting its own edge in any cycle it was part of.
  • Fall back to a controlled state. The waiter returns to a known safe restart point rather than continuing from a half-acquired mess, so the retry begins cleanly.
  • Retry under backoff. Reattempt after a randomized, escalating delay so simultaneous timeouts desynchronize instead of re-colliding.

Tuning parameters

  • Timeout threshold — how long a wait is tolerated before abandonment. Too short and healthy-but-slow waits are killed needlessly (false deadlocks); too long and real deadlocks sit unresolved.
  • Backoff schedule — fixed, linear, or exponential delay between retries, and how much random jitter is added. Jitter is the primary defense against livelock; too little and retries resynchronize.
  • Retry ceiling — maximum attempts before escalating to a person or a harder recovery. Without a ceiling, a persistent deadlock becomes an infinite thrash.
  • Fallback depth — how far back the "controlled state" is: just release-and-reattempt, or roll all the way to a saved checkpoint. Deeper fallback is safer but wastes more redone work.

When it helps, and when it misleads

Its strength is that it is detection-free and decentralized: no global graph, no coordinator, no authority — each participant self-heals on a clock, which scales to systems where building a wait-for graph is impossible or too slow. It degrades gracefully on transient stalls that aren't even true deadlocks, since a timeout resolves those too.

Its signature failure is livelock: participants that time out and retry in lockstep keep colliding, so the system is busy but frozen — motion without progress.[1] It also can't tell a deadlock from a merely slow operation, so an over-tight timeout abandons work that would have completed, wasting the effort and sometimes making load worse under contention. And without a clean fallback, repeated abandonment loses partial work or thrashes shared state. The discipline is randomized exponential backoff to break lockstep, a retry ceiling that escalates rather than loops forever, and a controlled fallback state so each retry starts clean instead of compounding the mess.

How it implements the components

Timeout and Retry Recovery fills the self-service break-and-resume components — no external actor required:

  • bounded_break_action — the expiry-triggered abandon-and-release is the intervention; bounded because the waiter only ever gives up its own attempt.
  • safe_state_checkpoint — the controlled state the waiter falls back to before retrying, so reattempts start from a coherent point.
  • participant_reentry_rule — the backoff-governed retry is the rule by which the abandoned participant re-enters and tries to progress again.

It performs no detection or mapping (deadlock_detection, cycle_map) — it substitutes a clock for diagnosis, unlike Wait-For Graph Analysis — and it involves no external authority (resolver_authority) or negotiated terms, which are the human-facing siblings' domain.

Notes

Timeout is the one break action that resolves deadlocks it never proves exist — which is both its economy and its blind spot. Because it can't distinguish deadlock from slowness, in a heavily contended system it can cause the pileup it's trying to clear (a "retry storm"). A tie-break criterion layered on top — always abandon the younger request, say — turns random re-collision into a decision, converting potential livelock into guaranteed progress; that is where it borders Tie-Break Rule.

References

[1] Livelock is the state where participants keep changing state in response to each other but none makes progress — the classic example being two people who repeatedly step the same way to let each other pass. It is the characteristic failure of naive timeout-and-retry, and randomized backoff is the standard cure. ↩