Skip to content

Deduplicating Message Consumer

Message-processing procedure — instantiates Asynchronous Replica Convergence

Remembers which message identities it has already processed so that a redelivered or duplicated message is recognized and dropped before it can repeat an effect.

Deduplicating Message Consumer sits on the receiving end of an at-least-once channel and guarantees each logical message takes effect at most once, by keeping a record of the identities it has already handled and checking every arrival against it. Its defining move is that deduplication happens at the consumer, on the effect side: even if the broker delivers the same event three times, the consumer performs the work once and swallows the rest. It does not stop duplicates from being produced or delivered — it neutralizes them at the boundary where side effects would otherwise fire.

Example

A billing service consumes payment.succeeded events from a broker that guarantees at-least-once delivery. After a transient network blip the broker redelivers the event for order #7781 twice. Naively, the consumer would credit the ledger twice and email two receipts. Instead it reads the event's identity — the payment's unique id — checks its store of already-processed ids, sees #7781 has been handled, and acknowledges the duplicate without re-running the ledger write or the receipt send. Crucially, the first copy's ledger write and its identity record commit together, so a crash mid-way can never leave the effect applied but the id unrecorded. The first copy did the work; every later copy is recognized and dropped.

How it works

The distinguishing method is identity-keyed suppression wrapped around the effectful step:

  • Extract an identity from each message — a broker message id or a business key that is stable across redeliveries.
  • Check the seen-set. If the identity is already recorded (within the retention window), acknowledge and drop; the effect does not run again.
  • Process atomically. For a new message, perform the effect and record its identity in a single atomic commit, so the two can never disagree.
  • Acknowledge last. Only after the effect-and-record commit is the message acked, so a failure before commit safely redelivers.

Tuning parameters

  • Identity source — broker-assigned message id versus a business key; the business key survives re-publishes the broker would treat as new messages, but must be genuinely unique per intent.
  • Dedup window — how long identities are retained; it must exceed the broker's maximum redelivery delay, or a late duplicate slips past an expired record.
  • Store durability — in-memory (fast, lost on restart) versus persistent (survives crashes, slower); sets whether dedup holds across a consumer restart.
  • Effect–record atomicity — whether the side effect and the identity record commit together; without it, a crash between them either double-applies or loses the guard.
  • Dedup scope — per-consumer versus a shared store across a consumer group; a shared store dedups across instances but adds a coordination point.

When it helps, and when it misleads

Its strength is converting an at-least-once channel into effectively-once processing at the one place it matters — the effect boundary — with nothing more than a remembered set of identities.[1] Its failure modes cluster around the window and the boundary: a duplicate arriving after the retention window has expired is treated as new, and any side effect that escapes the guarded boundary (an email already dispatched to an external system) cannot be recalled by dropping the duplicate. The classic misuse is trusting a broker's advertised "exactly once" that is really at-least-once under the hood, or letting the seen-set grow without bound until it becomes the system's memory leak. The discipline is to size the window to the real redelivery horizon, make effect-and-record atomic, and keep every externally-visible effect inside the boundary the guard protects.

How it implements the components

  • idempotence_guard — the seen-set of processed identities is the guard: a message whose identity is already recorded is a no-op, so reprocessing changes nothing.
  • side_effect_boundary — it gates the effectful step on that guard, so the observable effects of a logical message fire at most once no matter how many copies arrive.

It does not mint the operation_identity_key it checks — that is Idempotency Keys; it does not make handlers order- and grouping-insensitive (commutative_operation_rule, associative_grouping_rule) — that is Event Sourcing with Commutative Handlers; and it does not detect a conflicting write to a shared record (state_consistency_guard) — that is Optimistic Concurrency Check.

Notes

Deduplication and commutativity solve two different halves of "messages arrive badly." This consumer handles duplication — the same message twice — by remembering identities. It does not handle reordering — messages out of sequence — which needs handlers whose result is order-independent, the job of Event Sourcing with Commutative Handlers. A robust pipeline usually wants both.

References

[1] At-least-once delivery — most durable brokers guarantee a message is delivered at least once (so a retry never loses data) but may deliver it more than once, leaving the receiver responsible for idempotency. A deduplicating consumer is the standard "idempotent consumer" realization of that responsibility. ↩