Skip to content

Backfill and Rebuild Job

Rebuild workflow — instantiates Access-Optimized Redundant Representation

Bulk-populates or regenerates a redundant copy over its whole history — to create it, or to repair it after drift or a derivation bug — re-running safely without disturbing live traffic.

Backfill and Rebuild Job is the bulk, historical build path for a redundant copy — the batch process that materializes a copy from scratch across the entire back-catalog, whether to bring a brand-new projection online or to restore one that has drifted or been corrupted. Its defining move is that it operates on all of history at once, not on the stream of new changes: where incremental synchronization only touches what changes from now on, a backfill re-derives the past. Because it runs against live systems, it is built to be idempotent and throttled — safe to re-run, safe to resume, and gentle enough on the source that production reads never feel it. It is the recovery and bootstrap mechanism, distinct from whatever keeps the copy current afterward.

Example

A bug in how a denormalized full_name field was derived (it dropped middle names) has left ≈50 million existing user rows carrying the wrong value. Fixing the derivation only corrects rows that change from now on; the historical rows stay wrong forever unless something reprocesses them. A rebuild job iterates the users table in bounded batches keyed by id range, re-derives full_name for each row, and writes the corrected value — idempotently, so re-running a batch after a crash yields the same result rather than compounding it, and throttled to a few thousand rows per second so the job doesn't starve production queries. It checkpoints progress by id, so an interruption resumes mid-run. When it finishes, a reconciliation check confirms the copy has converged with the source.

How it works

  • Iterate history in bounded batches. Walk the source in checkpointed chunks and re-derive the copy for each, so progress is resumable, not all-or-nothing.
  • Run idempotently and throttled. Re-applying a batch yields the same result, and a throttle keeps the bulk work from crowding out live traffic.
  • Cut over to incremental sync. Rebuild to a consistent point, then hand ongoing maintenance to the steady-state sync so the copy stays current.

Tuning parameters

  • Batch size — larger batches finish faster but load the source harder; the dial trades rebuild time against production impact.
  • Throttle / backpressure — how aggressively to pace the job; the guardrail that stops a rebuild from taking down the source it reads.
  • Full rebuild vs. targeted repair — reprocess everything, or only the ranges a reconciliation check flagged as diverged.
  • Cutover strategy — dual-run the rebuilt copy alongside the old and switch, vs. rebuild in place; sets how much risk the switch carries.

When it helps, and when it misleads

Its strength is that it is the only way to bootstrap a new copy over existing data and the backstop for drift a sync pipeline can't self-heal — the path back to a correct copy. It misleads when run carelessly: un-throttled, a rebuild can overwhelm the source it reads from; non-idempotent, a retried batch double-applies; and if it races the live change stream it can overwrite newer data with a stale re-derivation.[1] The classic misuse is reaching for a full rebuild to paper over a synchronization bug — which just re-introduces the same drift on the next cycle. The discipline is idempotent, throttled, checkpointed jobs; fixing the root cause before rebuilding; and verifying convergence with a reconciliation check afterward.

How it implements the components

Backfill and Rebuild Job realizes the recovery-and-lifecycle side of the archetype — creating and restoring copies over their whole history:

  • recovery_and_reconciliation_path — it is the path back to a correct copy: bootstrap a new one, or restore a diverged or corrupted one from the source.
  • projection_lifecycle_rule — it operates the build and rebuild phases of a copy's life, distinct from the steady-state serving and sync that follow cutover.

It does not detect the drift that triggers it (that's Checksum and Sample Reconciliation), keep the copy current after cutover (Change-Data-Capture Propagation or Synchronization Job does), or define the copy's freshness bound (Freshness Watermark).

Notes

A rebuild must reconcile with the live write stream at cutover. A row that changes during the backfill can be silently lost if the job writes its stale re-derived value after a newer live update has already landed. Process from a consistent snapshot and arrange for incremental sync to win for anything newer than that snapshot, or the rebuild meant to fix drift will quietly create it.

References

[1] Idempotency — the property that applying an operation once or many times yields the same result. It is what makes a bulk job safe to retry and resume after failure; without it, every crash-and-retry risks double-counting, which is why rebuild jobs are written to converge rather than to accumulate. ↩