Skip to content

Replica-Lag and Freshness Dashboard

Telemetry dashboard — instantiates Shared-State Consistency Contract Design

Continuously measures how far replicas lag and how stale served reads actually are, and alerts when observed freshness breaches its promised bound or SLO.

Replica-Lag and Freshness Dashboard is the surface that answers one question in real time: is the consistency contract actually being met right now? It consumes raw signals — replication offset, version divergence, clock offset — and turns them into a freshness view: how far each replica lags, how stale the reads being served actually are, and what fraction land inside the promised staleness bound. Then it checks those measurements against a service-level objective and alerts (or triggers remediation) when the objective burns. What makes it distinct is that it measures the promise without making or keeping it: it neither serves reads nor enforces a bound, it holds the observed-versus-promised comparison so an abstract "eventually consistent" becomes a number someone is accountable for.

Example

A market-data service promises that quote reads served from replicas are no more than two seconds stale. The dashboard tracks per-replica replication lag, the p99 staleness of actually-served reads (measured with periodic canary writes whose read-back age it times), and a rolling "percentage of reads within the 2s bound" SLO. During a deploy, one replica falls forty seconds behind; the served-staleness p99 crosses two seconds and the freshness SLO's error budget starts burning. The dashboard alerts on-call and its remediation hook drains that replica from the read pool before the lag reaches customers. Nominal replication lag on the other replicas looked fine — it was the served-read freshness measurement, compared against the stated bound, that caught it.

How it works

The distinguishing move is measuring the client-perceived promise, not just server internals. It ingests lag and divergence signals emitted elsewhere, computes observed freshness distributions (ideally from canary reads that experience the real read path), and compares them to the contract's staleness bound — then expresses the result as an SLO with an error budget so that "within two seconds" has a defined tolerance and an alerting threshold. It is a measurement-and-comparison layer; the raw signals come from the replication layer and from read repair.

Tuning parameters

  • Freshness metric — lag-seconds, versions-behind, or percent-within-bound: which one clients actually feel, versus which is easy to collect.
  • SLO target & window — the freshness objective and its evaluation window / error budget, trading sensitivity against alert fatigue.
  • Measurement method — synthetic canary writes (accurate, some overhead) vs. passive lag scraping (cheap, less faithful to served reads).
  • Auto-remediation hook — whether an SLO breach drains a replica automatically: fast recovery vs. flapping risk.
  • Aggregation granularity — per-replica, per-key, or per-region rollups, trading detail against noise.

When it helps, and when it misleads

Its strength is making an abstract guarantee concrete and accountable: it gives early warning before clients notice, and ties consistency to an error budget the team can reason about.[1]

It misleads when a green dashboard is mistaken for correctness. Freshness is a symptom metric — it will not catch a lost write or a reordering that read repair never sampled, so a system can be on-target for staleness and still wrong. Vanity risk is real: average lag can look healthy while tail reads are badly stale. The backwards move is choosing the metric that looks good rather than the one clients feel, and letting "we monitor it" stand in for "we guarantee it." The discipline is to measure client-perceived freshness with canaries, and to pair the dashboard with a correctness oracle rather than relying on lag alone.

How it implements the components

  • service_level_objective — defines the freshness/lag objective, its error budget, and the alerting/remediation thresholds the contract is held to.
  • freshness_and_staleness_bound — measures observed staleness of served reads against the promised bound, making "how fresh, really?" a reported number.

It does not emit the raw divergence/lag signal itself (consistency_telemetry_signal) — that comes from Read Repair and the replication layer — nor enforce a policy that refuses too-stale reads (Bounded-Staleness Read Policy) or check history legality (Consistency History Checker).

References

[1] Service-level objective / error budget (SRE practice) — a target on a measured quantity, here freshness, with a bounded allowance for breach. Expressing consistency as an SLO is what turns "eventual" into something operationally testable and alertable. ↩