Skip to content

Replay Log or Event Stream

Durable event-log substrate — instantiates Topic-Brokered Event Distribution

Retains published events as an ordered, append-only log so any consumer can read — or re-read — from a chosen point, turning the event history itself into a replayable source of truth.

A Replay Log or Event Stream keeps the events a topic has carried as an ordered, append-only log rather than discarding each one the moment it is delivered. Every consumer reads the same log at its own position, and — crucially — can rewind that position to re-read history from an earlier point. Its defining move is to decouple producer and consumer in time as well as in identity: a subscriber does not have to exist, or be healthy, at the moment an event is published, because the event is still there to be read later. Where the rest of the archetype routes events to whoever is listening now, this component preserves them for whoever needs them next.

Example

A factory runs a fleet of vibration and temperature sensors publishing telemetry to a stream retained for 180 days. Months in, a data team wants to stand up a predictive-maintenance model, which needs roughly ninety days of history to train and to warm up its rolling baselines. Because the telemetry lives on a replay log, the new consumer simply starts reading from an offset ~90 days back, replays the retained history at full speed until it catches up to the live edge, then continues in real time — no producer change, no one-off backfill export, no coordination with the teams already consuming the same stream. Had the telemetry gone through a transient, delete-on-delivery queue, that history would simply be gone, and the model could only ever learn from the future.

How it works

  • Append-only ordering — events are written once, in order, and never mutated; a consumer's position is just an offset into that sequence.
  • Independent read positions — each consumer (or group) tracks its own offset, so replaying for one does not disturb the delivery any other is receiving.
  • Bounded retention — a policy decides how long history survives: a time window, a size cap, or compaction that keeps only the latest event per key.
  • Domain-scoped logs — the stream is drawn around one bounded event domain, so its retention, ordering, and access rules are coherent rather than a mixture of unrelated concerns.

Tuning parameters

  • Retention window — how far back history reaches; longer enables late consumers and audits but grows storage without bound if left unchecked.
  • Compaction — keep the full history of every event, or only the latest value per key; compaction shrinks the log but destroys the intermediate story.
  • Replay throughput cap — how fast a rewinding consumer may re-read; uncapped replay can saturate the broker and starve live readers.
  • Ordering scope — global order (one partition, limited throughput) versus per-key order across many partitions (scalable, but only ordered within a key).
  • Offset checkpoint granularity — how often a consumer commits its position, trading redelivery-on-restart against bookkeeping overhead.

When it helps, and when it misleads

Its strength is that new and recovered consumers can rebuild their own state from the record, audits can reconstruct exactly what was published, and a bug fixed today can be re-run against last week's events. Its failure mode is treating unlimited retention as a free crutch: storage grows without end, and replaying old events re-fires their side effects — a rewind can re-send yesterday's emails if consumers are not idempotent. The classic misuse is querying the log as though it were a database, seeking randomly through history for point lookups it was never shaped to serve. The discipline is to bound retention deliberately, keep consumers idempotent[1], and mark replayed events so side-effecting handlers can tell a rewind from live traffic.

How it implements the components

  • replay_and_retention_policy — it is the retained, re-readable history; the log's whole reason for existing is to hold events past delivery and serve them again from any offset.
  • event_domain_boundary — each log is scoped to one bounded event domain, giving that domain a single ordered spine with coherent retention and access rules.

It preserves and re-serves events but does not route them to matching subscribers (broker_routing_boundary — Topic Exchange or Event Bus), hold a private backlog for one offline subscriber (that's Durable Subscription Queue), or confirm each subscriber actually processed an event (delivery_trace_record — that's Delivery Acknowledgement).

Notes

A replay log and a durable subscription queue are easily confused but solve opposite problems: the queue is a private backlog holding one subscriber's undelivered messages until it returns, and drains as it is consumed; the log is a shared record every consumer reads independently and no one drains. Reach for the log when history has value beyond its first delivery; reach for the queue when a specific subscriber just needs to not miss messages while it is away.

References

[1] Event sourcing — the pattern of treating an append-only log of events as the authoritative source of truth, from which current state is derived by replaying them. It is what makes a replay log more than a buffer, and it is also why idempotent, replay-aware consumers are a precondition rather than a nicety: state is rebuilt by re-reading, so re-reading must be safe. ↩