Skip to content

Scheduled Incremental Refresh

Scheduled refresh process — instantiates Access-Optimized Redundant Representation

On a fixed cadence, applies only the source changes since the last run to a redundant copy, keeping it current to a bounded lag without the cost of a full rebuild.

A Scheduled Incremental Refresh keeps a redundant copy current by running on a clock and, each run, applying only what changed since the previous run rather than rebuilding from scratch. It reads a marker of where it last left off, pulls the source rows that changed after that point, applies them to the copy, and advances the marker. Its defining pair of choices is cadence and delta: the schedule sets the copy's worst-case staleness (a copy refreshed every ten minutes is never more than ten minutes behind), and the incremental scope keeps each run cheap by touching only the changed slice. It is the middle path between event-driven propagation, which is continuous but complex, and a full rebuild, which is simple but expensive — periodic, bounded, and self-limiting in cost.

Example

A warehouse-management system runs picking and receiving against a busy transactional database. Floor staff need a read model of on-hand quantity by location, but pointing their tablets at the live operational tables both slows fulfilment and gives inconsistent counts mid-transaction. Real-time streaming would be overkill — a few minutes of lag is perfectly acceptable for a "roughly how much is where" view.

They schedule an incremental refresh every ten minutes. Each run reads the last-processed transaction timestamp, selects only the inventory movements committed since then, folds them into the read model's per-location counts, and advances the timestamp. A run touches a few thousand changed rows, not the whole catalog, so it finishes in seconds. The read model is never more than ten minutes stale — a bound staff can rely on — and the operational database is spared a constant read load. When a run fails, the marker is not advanced, so the next run simply picks up the same delta plus a little more; nothing is skipped.

How it works

  • Read the low-water mark. Start from the position the last successful run reached — a timestamp, a change-log offset, or a version column — so the run knows exactly which source changes are new.
  • Pull and apply only the delta. Select the source rows changed since that mark and merge them into the copy, rather than recomputing the whole copy; cost scales with churn, not with total size.
  • Advance the mark atomically on success. Move the marker forward only once the delta is safely applied, so a failed or partial run is simply retried from the same point — the process is restart-safe by construction.

Tuning parameters

  • Cadence — how often the job runs. Tighter cadence lowers the staleness bound and shrinks each delta, but raises scheduling overhead and load on the source; looser cadence is cheaper but staler.
  • Delta detection — timestamp column, monotonic version, or a change log. Timestamps are simple but miss out-of-order or clock-skewed updates; a change log is exact but needs the source to emit one.
  • Batch window vs. micro-batch — large infrequent runs versus small frequent ones. Micro-batching smooths load and lowers lag; large batches amortize fixed per-run costs but spike resource use.
  • Failure policy — skip, retry, or widen the next window. Never advancing the mark on failure guarantees no lost changes at the risk of reprocessing; advancing regardless risks silent gaps.

When it helps, and when it misleads

Its strength is a predictable, self-limiting way to hold a copy to a stated staleness bound at a cost proportional to change, not to size. When some lag is acceptable and a full rebuild would be wasteful, it is the pragmatic default — simpler to reason about than streaming, far cheaper than rebuilding.

It misleads chiefly through incremental drift: apply only deltas forever and small errors — a missed out-of-order update, a delete the timestamp scheme never saw, a bug in one run — accumulate invisibly, so the copy slowly diverges from source in ways no single run reveals. This is the well-known limit of incremental view maintenance[^ivm]: it is efficient but not self-correcting. Timestamp-based delta detection is the usual culprit, silently missing updates that land with an earlier timestamp than the mark. And a fixed cadence cannot promise freshness tighter than its interval, so it is the wrong tool when a reader needs their own write reflected instantly. The discipline is to pair incremental refresh with a periodic full rebuild or reconciliation that re-establishes ground truth, choose a delta-detection scheme that cannot miss changes, and never sell a cadence-bounded copy as real-time.

How it implements the components

Scheduled Incremental Refresh supplies the keep-it-current components — the cadence and delta mechanics, not the copy's shape or content verification:

  • freshness_rule — the schedule is the freshness policy: it governs how often the copy is brought current and therefore how fresh readers can expect it to be.
  • synchronization_rule — it defines how the copy tracks source: read the mark, apply the delta since it, advance the mark, restart-safe on failure.
  • freshness_and_staleness_bound — the cadence sets the copy's worst-case staleness; the interval is the bound the rest of the design can rely on.

It does not observe or publish the resulting freshness (version_token, divergence_signal) — that is Freshness Watermark — and it cannot re-establish ground truth after drift; a full Backfill and Rebuild Job or Reconciliation Workflow does that.

Notes

Incremental refresh and a periodic full rebuild are complements, not alternatives: the incremental job keeps the copy cheap and fresh day to day, while an occasional rebuild or reconciliation erases the drift that incremental application inevitably accretes. Running only the incremental job is the most common way a copy quietly rots.

References

Incremental view maintenance is the established technique of updating a derived view by applying only the changes to its inputs rather than recomputing it. It is efficient but not self-correcting — missed or out-of-order changes accumulate — which is why it is paired with periodic full recomputation.