Replica Repair Job¶

Background repair process — instantiates Asynchronous Replica Convergence

Runs on a schedule to find replicas that have fallen behind or diverged and reconciles them back toward the others, bounding how stale any copy is allowed to get.

Replica Repair Job is the background process that actively hunts divergence and fixes it, rather than waiting for a read to expose it. On a schedule — or when a replica rejoins after a partition — it compares a replica against its peers, finds what it is missing or holds wrongly, and streams the corrections in, enforcing a ceiling on how far any copy may drift. Its defining move is scheduled, proactive convergence under a staleness bound: it guarantees that even cold data nobody is reading gets reconciled within a known window, converting "eventually consistent" into "consistent within N."

Example¶

A five-node key-value store loses node C to a two-hour network partition. During the split, the other nodes accept writes C never sees. When C rejoins, a repair job kicks in: it compares C's key ranges against healthy replicas — using a divergence scan to locate only the differing ranges cheaply — then streams the missing and superseded records into C until its ranges match. Independently, a staleness threshold ("no replica more than four hours behind") triggers periodic repairs even absent any failure, so drift from occasional dropped messages never accumulates past the bound. Throttling keeps the repair traffic from starving live requests. Once C is within tolerance it is marked healthy again — and, crucially, the convergence never depended on a client happening to read the stale keys.

How it works¶

The distinguishing method is orchestration and a bound, sitting above the low-level reconciliation:

Trigger. Run on a schedule, on replica rejoin, or when a replica breaches the staleness bound.
Locate divergence. Obtain a signal of which ranges or records differ, rather than re-shipping everything.
Repair from a trustworthy source. Fetch the newer or missing records from peers (ideally quorum-agreed), applying the system's merge and tombstone rules.
Confirm and reset the clock. Verify the range now matches and reset the replica's staleness measure — and, when repair volume climbs, treat that as a symptom to diagnose.

Tuning parameters¶

Repair cadence / trigger — periodic, rejoin-driven, or staleness-driven; more frequent repair shrinks drift but raises constant load.
Staleness threshold — how far behind any replica may fall before repair is forced; tight bounds keep data fresh but demand more repair work.
Scope and throttle — whole replica versus hot ranges, plus a bandwidth cap so repair never starves live traffic.
Repair source — repair from a single peer (cheap, risks copying bad data) or from a quorum / verified source (safer, costlier).
Priority — which ranges or keys are reconciled first when a replica is far behind.

When it helps, and when it misleads¶

Its strength is bounding worst-case staleness for all data — including cold data that on-read repair never touches — and healing partitions proactively instead of waiting for a reader to stumble on the gap.^[1] Its failure modes come from cost and trust: an aggressive schedule can saturate the network and starve live traffic in a "repair storm," repairing from a wrong source can propagate corruption instead of fixing it, and a too-loose threshold lets drift linger unseen. The classic misuse is cranking the cadence to mask an underlying propagation bug — repairing constantly rather than fixing why replicas drift. The discipline is to throttle against live load, repair only from a trustworthy (quorum- or scan-verified) source, and treat rising repair volume as a diagnosis to make, not merely a cost to absorb.

How it implements the components¶

update_propagation_and_repair_contract — it is the contract for bringing a lagging replica current: what triggers repair, which source counts as authoritative, and how corrections are applied.
staleness_threshold — the explicit bound on how far behind any replica may fall before repair is forced, which is what turns "eventually" into "within a known window."

It orchestrates repair but does not perform the low-level pairwise reconciliation exchange (Anti-Entropy Reconciliation Exchange), compute which ranges differ (drift_detection_signal, Merkle-Tree Divergence Scan), or fix a stale value opportunistically at read time (Read Repair on Access).

Instantiates: Asynchronous Replica Convergence — it is the scheduled guarantee that update frontiers actually meet, so replicas cannot look healthy while silently diverging.
Consumes: a divergence scan (Merkle-Tree Divergence Scan) to locate differences and a reconciliation exchange (Anti-Entropy Reconciliation Exchange) to move the corrected records.
Sibling mechanisms: Data Diff and Merge Tool · Exception Queue Review · Anti-Entropy Reconciliation Exchange · Merkle-Tree Divergence Scan · Read Repair on Access · Hinted-Handoff Buffer

Notes¶

It is easy to confuse with its two repair siblings, but the distinction is when and what it covers. Read Repair on Access fixes only the data a read happens to touch; Anti-Entropy Reconciliation Exchange is the pairwise protocol that actually moves the bytes. This job is the scheduled orchestration with a staleness bound that decides which replicas to repair and when — the only one of the three that guarantees cold, unread data is reconciled within a fixed window.

References¶

[1] Anti-entropy — the family of background processes that continuously compare replicas and repair differences so divergence cannot accumulate, named by analogy to fighting the natural drift toward disorder. It is the proactive complement to fixing staleness only when a read happens to expose it. ↩