Skip to content

Hinted-Handoff Buffer

Stand-in write buffer — instantiates Asynchronous Replica Convergence

When a replica is unreachable, parks the writes meant for it on a stand-in node and replays them the moment it returns, so a brief outage neither loses nor blocks updates.

Hinted-Handoff Buffer is the fast bridge across a single replica's temporary absence. When a write is meant for a replica that is momentarily unreachable — rebooting, deploying, briefly partitioned — a healthy node accepts the write anyway and stores it as a hint: a held-aside copy tagged "for replica N." The moment N returns, the hints are replayed to it and deleted. Its defining move is to key repair on readiness and act on the write path: it watches which replica is down and keeps that node's would-be writes moving forward on its behalf, so the outage costs neither availability nor a lost update. It is a bounded bridge, not a backstop — hold a hint only for a set window, then let the background sweep take over.

Example

A messaging service replicates each conversation across three nodes. Mid-deploy, node 3 — a replica for Priya's chat — is restarting when she sends a message. Rather than reject the write or block on the missing node, the coordinator writes the message to a healthy stand-in and marks it a hint for node 3. Priya's message is delivered and durable on the live replicas immediately; she notices nothing.

Ninety seconds later node 3 rejoins, the stand-in replays the buffered hints to it, and node 3 is caught up without waiting for the next background reconciliation cycle. Had node 3 stayed dark past the buffer's time-to-live — say ≈3 hours — the stand-in would drop the hints and leave node 3's catch-up to full anti-entropy. A brief outage became invisible, and the buffer quietly disappeared once its job was done.

How it works

What distinguishes it from the other repair mechanisms is targeting and timing. It is driven by readiness: it acts precisely when a specific replica is detected down, and stops when that replica is back. It works on the write path, not by scanning or reading — it captures the exact writes that would have gone to the absent node and replays them on return. And it is deliberately bounded: a hint has a time-to-live, so if the node stays gone too long the buffer gives up rather than growing without limit, deferring to the exhaustive background sweep. Replayed hints are made safe against duplication by an idempotence guard it consumes, so a partially-delivered replay never double-applies.

Tuning parameters

  • Hint time-to-live — how long to hold a downed replica's writes before dropping them and deferring to background repair. Longer bridges longer outages but risks an unbounded buffer; shorter caps memory but hands more work to the slow path.
  • Buffer durability — whether hints live in memory or are persisted. Persisted hints survive the stand-in's own crash (closer to a real guarantee); in-memory hints are faster but evaporate if their holder dies.
  • Down-detection sensitivity — how quickly a replica is judged "unavailable" enough to start buffering. Trigger too eagerly and you buffer through transient blips; too slowly and early writes are rejected.
  • Stand-in selection — which healthy node holds another's hints. Spreading hints avoids a single overloaded stand-in.

When it helps, and when it misleads

Its strength is turning the common case — a node down for seconds or minutes during a deploy, reboot, or blip — into a non-event, keeping the write path available and letting the returning replica catch up in one cheap replay instead of a full reconciliation. It is what lets a system accept writes at all during a partial outage.[1] But hints are best-effort, not durable replication: if the stand-in holding them fails before replay, or a node stays down past the time-to-live, those hints are gone and convergence falls back to the background sweep. The classic misuse is counting a hint toward the replication factor — acking a write as "safely replicated" when really only one live node holds it and a hint holds the rest — so a stand-in failure loses acknowledged data. The discipline is to treat hinted-handoff as a bridge that complements durable replication and anti-entropy, never a substitute, and to bound the buffer so a long outage degrades gracefully to the slow path.

How it implements the components

Hinted-Handoff Buffer fills the readiness-and-bounds components — the ones a write-path bridge for a downed replica can fill:

  • synchronization_or_readiness_state — it tracks each replica's up / down / recovered state and switches between buffering and replaying on that signal.
  • divergence_envelope — the hint time-to-live is the bounded window it will cover: how long and how far a replica may fall behind before the buffer gives up and the background sweep takes over.

It does not run the exhaustive background reconciliation (that is Anti-Entropy Reconciliation Exchange) or make its own replays safe against duplication (that is Idempotency Keys, which it consumes). It bridges one replica's absence, not the whole keyspace.

References

[1] In a sloppy quorum, writes meant for a temporarily unreachable replica are accepted by substitute nodes and later delivered to the intended replica via hinted handoff, preserving write availability during faults. The hint is explicitly a temporary stand-in for the absent replica, not an extra durable copy — which is why it must eventually hand off or expire rather than be counted as replication. ↩