Skip to content

Read Repair

Read-path repair procedure — instantiates Shared-State Consistency Contract Design

On each read, compares the versions the queried replicas return and writes the freshest value back to the stale ones, healing divergence opportunistically on the read path.

Read Repair is a reconciliation procedure that piggybacks on reads. When a read gathers responses from several replicas and they disagree, the newest version is returned to the client and pushed back to the replicas that were behind, so the divergence is healed as a side effect of traffic that was happening anyway. What makes it distinct among the archetype's mechanisms is its trigger: repair is driven by reads, not by a background sweep (anti-entropy) or by resolution at write time (multi-leader). This makes it demand-proportional — the data people actually read stays fresh cheaply — and it doubles as a sensor: every read that finds replicas disagreeing is a live sample of how divergent the system currently is.

Example

A social service stores follower counts in a wide-column store such as Apache Cassandra with three replicas per key. One replica briefly missed a recent increment. A user loads a popular profile; the read is served at a consistency level that consults two or three replicas, and the coordinator notices version 41 on two replicas but version 40 on the third. It returns 41 to the user and, in the background, writes 41 back to the lagging replica — which is now healed without any operator action. Because the profile is popular, this happens within seconds of the first read; a cold profile that nobody views would stay divergent until a scheduled repair reaches it. The stream of "replicas disagreed on this read" events is also exported as a divergence signal, quietly reporting how much staleness the fleet is carrying.

How it works

The distinguishing mechanics are that repair is opportunistic and read-scoped. It compares versions returned by a read (often via lightweight digests rather than full values to save bandwidth), determines the winner, and back-writes it to the laggards — either blocking (repair before the read returns, for stronger convergence) or asynchronous (return first, repair after, for lower latency). Because it only touches keys that are read, it converges hot data fast and never reaches cold data on its own, which is why it is paired with, not a substitute for, a background reconciliation pass.

Tuning parameters

  • Blocking vs. asynchronous — repair before returning (stronger, slower) or after (faster, briefly non-monotonic).
  • Repair probability — the chance of repairing on lower-consistency reads: higher coverage for more overhead.
  • Digest vs. full read — compare hashes to save bandwidth, or ship full values to cut round-trips.
  • Read breadth — how many replicas a read consults: more replicas catch more divergence but cost latency.
  • Coupling with anti-entropy — how much cold-data convergence is left to background repair, trading read-path cost against completeness.

When it helps, and when it misleads

Its strength is nearly-free convergence for read-hot data and a continuous, no-extra-cost divergence signal that surfaces replication health.

It misleads when treated as a completeness guarantee. Read repair only heals data that gets read — cold, rarely-accessed keys can stay silently divergent indefinitely, so a system that relies on it alone will rot in its long tail.[1] Asynchronous repair can also briefly show a client an old value and then a new one (a non-monotonic flicker) unless a session guarantee is layered on. The classic mistake is skipping scheduled anti-entropy because "read repair covers it" — leaving exactly the unread data unrepaired. The discipline is to treat read repair as opportunistic acceleration and always pair it with a background pass for the cold set.

How it implements the components

  • recovery_and_reconciliation_path — the back-write of the freshest value to lagging replicas is the reconciliation path, triggered on the read path.
  • consistency_telemetry_signal — each read that finds replicas disagreeing is emitted as a raw divergence/repair-rate sample.

It does not build dashboards or SLOs over that signal (service_level_objective, freshness_and_staleness_bound) — that is Replica-Lag and Freshness Dashboard — converge unread keys (Anti-Entropy Synchronization), or decide which replicas a read consults (read_selection_rule, Quorum Read/Write Protocol).

References

[1] Anti-entropy — background reconciliation (e.g., Merkle-tree comparison between replicas) that converges data independent of reads. Read repair is its read-triggered special case: cheaper, but complete only over the set of keys actually read. ↩