Skip to content

Change-Data-Capture Propagation

Change-propagation pipeline — instantiates Access-Optimized Redundant Representation

Tails the source database's commit log and streams every row-level change to downstream copies, so they follow the source in near real time without the application having to dual-write.

Change-Data-Capture Propagation keeps redundant copies current by reading the source database's own commit log — the write-ahead log or binlog — and turning each committed insert, update, and delete into an ordered stream that downstream copies apply. Its defining move is where the changes come from: not from the application code, and not from database triggers, but from the log the database already writes for durability. That makes capture complete (every committed change is there), correctly ordered, and nearly free on the source, and it sidesteps the dual-write problem in which an app tries to update the primary and a copy separately and the two can diverge on any partial failure. CDC is the asynchronous propagation backbone that fans one source's changes out to indexes, caches, warehouses, and read models.

Example

A retail catalog's authoritative product data lives in an OLTP database; a search index and a cache both need to reflect edits within seconds. Rather than have the catalog service write to all three (and risk one succeeding while another fails), a log-based CDC connector — Debezium tailing the binlog, say — emits an ordered stream of row changes. Two consumers subscribe: one re-indexes the changed product in search, the other invalidates the affected cache keys. A price edit committed at 10:00:00 appears in search by ≈10:00:03, and the stale cache entry is dropped so the next read repopulates it. The catalog service's write path is untouched; it just writes to its own table as before, and the log does the fan-out.

How it works

  • Capture from the log. A connector reads committed changes from the database's transaction log, so capture is complete and in commit order without app involvement.
  • Publish as an ordered stream. Each change becomes an event on a durable stream that many independent consumers can read at their own pace.
  • Apply or invalidate downstream. Consumers update their copy from the change, or mark the affected entries stale so they refresh on next read.

Tuning parameters

  • Capture method — log-based (lowest source overhead, complete) vs. query- or trigger-based (simpler, but adds load and can miss deletes).
  • Delivery semantics — at-least-once (needs idempotent consumers) vs. exactly-once (stronger, costlier); sets what consumers must handle.
  • Propagation lag budget — how far behind real time copies may run; tighter lag needs more throughput headroom.
  • Snapshot + stream — whether a new consumer first backfills an initial snapshot and then tails, so it starts from a consistent point.

When it helps, and when it misleads

Its strength is keeping many copies fresh with almost no burden on the source and no change to application write paths — one capture, arbitrary fan-out. It misleads when consumers assume guarantees the stream doesn't give: strict global ordering across partitions, or exactly-once without their own deduplication.[1] It also breaks quietly when the source schema changes underneath it and the stream's shape shifts. The classic misuse is treating row-level change events as if they were business events — they are deltas of storage, not domain facts, and reconstructing intent from them is fragile. The discipline is idempotent consumers, an explicit initial snapshot before tailing, and versioning the change schema so a source migration doesn't silently corrupt every copy.

How it implements the components

Change-Data-Capture Propagation realizes the synchronization side of the archetype — the transport that keeps copies following the source:

  • synchronization_rule — it propagates every committed source change to the copies, defining how and in what order they follow.
  • freshness_rule — continuous log-tailing governs how quickly a change reaches a copy, holding copies to a near-real-time freshness policy.
  • invalidation_rule — each captured change marks the affected downstream entries stale or to-update, driving cache and index invalidation.

It does not define the target representation it feeds (Materialized View or CQRS Read-Model Projection do), update a copy synchronously inside the writing transaction (that's Database-Trigger Synchronization), or repair a copy that has already diverged (that's Reconciliation Workflow or Backfill and Rebuild Job).

Notes

CDC can only tail changes from now; it cannot invent the past. A brand-new copy must first be seeded with a consistent snapshot (Backfill and Rebuild Job) up to a known log offset, and the stream applied from exactly that offset. Skip the snapshot, or overlap it sloppily with the stream, and the copy carries a permanent gap or a double-apply that no amount of later streaming will heal.

References

[1] The dual-write problem — when an application writes the source and a copy in two separate operations, any partial failure leaves them inconsistent with no single log of what actually happened. Log-based CDC avoids it by deriving all copies from the one commit log the database already maintains, so there is a single ordered source of change. ↩