Skip to content

Event Replay Deduplication

Idempotency tool / consumer-side control — instantiates Event-Log-Centered Modeling

Lets a consumer process an at-least-once event stream safely by keying on stable event identifiers, so a redelivered or replayed message never applies its effect twice.

Real event streams deliver at least once, not exactly once: on retries, failovers, and replays, the same event will arrive again. Event Replay Deduplication is the consumer-side machinery that makes this harmless. It gives every event a stable identity key, remembers which keys it has already applied, and drops or short-circuits any event it has seen before — so processing the same event twice has the same effect as processing it once. Its defining concern is idempotent application: not preventing duplicates from being delivered (impossible in a distributed system) but preventing them from being applied. This is precisely what makes replay and retries safe to run against side-effecting consumers — the very thing that would otherwise turn a routine event replay into a second round of charges, shipments, or emails.

Example

A notification service consumes an event stream and sends push notifications: OrderShipped → "Your order is on its way." The message broker guarantees at-least-once delivery, so during a broker failover the service is handed a batch it had already processed. Without deduplication, every customer in that batch gets a duplicate "on its way" ping — a small disaster at scale.

The dedup layer prevents it. Each event carries a stable identity key (here, a shipment_id chosen so retries of the same logical shipment share the key, while genuinely distinct shipments never collide). Before sending, the consumer checks the key against its record of already-notified shipments; the replayed batch's keys are all present, so every one is dropped and no duplicate pings go out. When a late event arrives — a DeliveryConfirmed that was delayed and shows up after a later status — the late-and-out-of-order policy decides its fate: apply it if it still moves the state forward, discard it if a newer event has already superseded it. The consumer's effects are the same whether the stream is delivered cleanly once or messily three times.

How it works

Deduplication lives entirely on the consuming side and turns on two levers:

  • Assign a stable identity key. Each logical event gets an identifier that is the same across every redelivery and different across genuinely distinct events — the whole scheme's correctness rides on choosing this key well.
  • Remember what's been applied. The consumer keeps a record (a processed-key set or a high-water mark) of the keys whose effects it has already committed, and consults it before acting.
  • Drop or short-circuit repeats. An event whose key is already recorded is suppressed, so re-application is a no-op.
  • Rule on late and out-of-order arrivals. A policy decides whether a delayed or reordered event is still applied, deferred, or discarded as already-superseded — because "already seen" and "arrived too late to matter" are different cases.

Tuning parameters

  • Identity-key choice — business key versus generated message id. A business key dedupes true logical duplicates but requires domain judgment; a generated id is mechanical but may miss semantic duplicates arriving by different paths.
  • Dedup window — how long processed keys are remembered. A long window catches far-apart duplicates but costs storage; a short window is cheap but lets old redeliveries slip through.
  • Late-event tolerance — how long after an event's time it may still be applied. Generous tolerance corrects more genuinely-late data; tight tolerance protects downstream state from surprise mutations.
  • Out-of-order handling — reorder-and-apply versus reject-if-stale. Reordering yields correct final state but adds buffering; reject-if-stale is simpler but can drop legitimate late corrections.
  • Effect atomicity — whether recording the key and applying the effect commit together. Atomic commit prevents the "applied but not recorded" gap; non-atomic is faster but reopens the duplicate window on a crash.

When it helps, and when it misleads

Its strength is that it makes at-least-once delivery usable: retries, failovers, and full replays become safe operations rather than hazards, which is what lets the rest of the archetype rebuild and reprocess freely. Idempotent consumers are the standard way real systems get "exactly-once effects" out of "at-least-once delivery."[1]

It misleads when the identity key is chosen badly. Too narrow a key lets true duplicates through (double charges); too broad a key collides distinct events and silently drops real ones — and the second failure is worse because it looks like nothing happened. Dedup also cannot fix a non-idempotent effect it doesn't mediate: if the downstream action isn't routed through the check, replay still double-fires. And a finite dedup window means a duplicate delivered after the window reopens the bug. The discipline is to pick the identity key from the event's semantics, commit the key-record and the effect atomically, and size the window to the real redelivery horizon rather than to convenience.

How it implements the components

Event Replay Deduplication realizes the idempotent-application side of the archetype — the components that make repeated and disordered delivery safe:

  • event_identity_key — it assigns and matches the stable key that identifies a logical event across all its redeliveries.
  • late_and_out_of_order_event_policy — it rules on whether a delayed or reordered arrival is applied, deferred, or discarded as superseded.

It supplies idempotency but not the log, the ordering, or the reconstruction: the canonical ordered store and its sequencing are Append-Only Event Store, and rebuilding state from the replayed events is Deterministic Replay Protocol.

References

[1] An idempotent consumer produces the same result whether it processes a message once or many times. It is the standard technique for achieving exactly-once effects on top of at-least-once delivery, which most message systems provide because true exactly-once delivery is impractical. ↩