Skip to content

Dead-Letter Queue

Failure-capture queue — instantiates Topic-Brokered Event Distribution

A side queue that captures events a subscriber cannot process after its retries are exhausted, isolating poison messages and preserving them as evidence instead of losing or looping them.

A Dead-Letter Queue (DLQ) is where an event goes when a subscriber has tried and failed to process it enough times to give up. Rather than redelivering a poison message forever — one that will fail on every attempt because it is malformed, references missing data, or trips a bug — the broker diverts it, after a redelivery limit, into a separate holding queue. Its distinguishing job is to be the evidence store for undeliverable events: each dead letter is retained with the reason it failed and how many attempts it took, so failures become an inspectable record instead of either silent loss or an infinite retry loop. The DLQ is the failure terminus of the delivery path — it does not retry, route, or acknowledge; it captures, isolates, and preserves.

Example

A notifications service consumes user.events to send emails. One event has a malformed recipient address that throws on every attempt. Under at-least-once delivery this is dangerous: the broker will redeliver the failing event again and again, and because a stuck message can hold up the ones behind it, one bad address threatens to stall the whole subscription — the poison-message trap.

The subscription is configured with a redelivery limit of five and a dead-letter queue. After the fifth failed attempt the broker stops retrying that event and moves it to the DLQ, stamped with the last error ("invalid recipient") and the attempt count. The live stream flows on — the poison message is out of the hot path. An on-call engineer later reads the DLQ, sees a cluster of invalid-address failures pointing at a bug in an upstream form, fixes the source, and replays the salvageable dead letters. Nothing was lost, nothing looped, and the failures became a diagnosis.

How it works

  • Trip on exhausted retries. When an event exceeds its redelivery limit (or is explicitly rejected as unprocessable), the broker diverts it instead of retrying again.
  • Capture with cause. Each dead letter is stored together with metadata — failure reason, attempt count, original topic, timestamps — so it is a record, not just an orphaned message.
  • Isolate the poison. Moving the failing event off the live queue stops it from blocking or endlessly reprocessing, containing one bad message's effect on the stream.
  • Await triage. Dead letters accumulate for human or automated inspection, with the option to fix the cause and replay, or to discard as genuinely unrecoverable.

Tuning parameters

  • Redelivery threshold — attempts before dead-lettering. Too low banishes transient failures that would have succeeded on retry; too high wastes cycles and delays isolation of true poison.
  • Capture scope — full payload plus rich failure context versus a lean pointer. Richer capture aids diagnosis but costs storage and may retain sensitive data.
  • DLQ retention — how long dead letters are kept before purging. Longer preserves more evidence but grows unbounded if no one triages.
  • Alerting threshold — what dead-letter rate or count raises an alarm. Sensitive alerting catches problems early but risks noise from routine one-offs.
  • Replay policy — whether dead letters can be re-injected after a fix, and with what de-duplication. Replay recovers data but can reintroduce duplicates if not guarded.

When it helps, and when it misleads

Its strength is converting three bad outcomes into one good one: instead of silent loss, infinite retry loops, or a poison message[1] stalling a subscription, failed events are isolated off the hot path and preserved as a diagnosable record. It is what makes at-least-once delivery survivable in the presence of messages that will never succeed.

It misleads when it becomes a place things go to be forgotten. An unwatched DLQ is a silent graveyard — real failures piling up while everyone assumes the pipeline is healthy because the live stream looks fine. Dead-lettering too eagerly (a low threshold) banishes events that a brief retry would have delivered, turning transient blips into manual work. And a DLQ tempts teams to treat symptoms, endlessly replaying dead letters without fixing the upstream cause that produced them. The discipline is to alarm on dead-letter rate, treat the queue as a work list with an owner rather than storage, and use its captured causes to fix sources instead of just re-injecting failures.

How it implements the components

  • delivery_trace_record — each dead letter is a durable record of a delivery that failed, stamped with cause and attempt count: the evidence trail of what could not be delivered and why.
  • backpressure_and_isolation_profile — by diverting poison messages off the live queue, it isolates one unprocessable event from the rest of the stream, preventing head-of-line blocking and retry storms.

It does not run the retry-and-ack protocol that precedes dead-lettering (Delivery Acknowledgement), retain the live backlog for an offline consumer (Durable Subscription Queue), route or fan events out (Message Broker), or validate structure so bad messages are rejected earlier (Schema Registry).

Notes

A dead-letter queue is only as useful as the attention paid to it. Because the live pipeline keeps flowing once poison is diverted, the DLQ's failures are invisible unless something surfaces them — so it should always be paired with alerting or a health dashboard. Treated as a monitored work list, it is a diagnostic asset; treated as a bucket, it is silent data loss with extra steps.

References

[1] A poison message is one that a consumer can never successfully process, so that under naive redelivery it is retried indefinitely and can block or destabilize the stream. Diverting such messages to a dead-letter channel after a retry limit is the standard remedy. ↩