Dead-Letter Queue Processing¶

A drain process for un-processable items — instantiates Queue Draining

Diverts messages that repeatedly fail processing into a separate queue where they can be inspected, corrected and retried, or deliberately discarded — so poison items never stall the main drain.

Some items in a queue simply cannot be processed — a malformed record, a reference to something deleted, a bug that throws every time. Left in place they are retried forever, blocking everything behind them. Dead-Letter Queue Processing is the mechanism that gets them out of the way: after a set number of failed attempts, an item is moved off the main path into a separate dead-letter queue, where a human or tool decides its fate — fix and replay, discard with a logged reason, or escalate. Its defining idea is failure-triggered diversion with preservation: the main drain keeps flowing past the poison item, and the item is neither lost nor infinitely retried but held aside, visible and recoverable.

Example¶

An order-processing pipeline consumes order placed events. One event references a product SKU deleted in a bad catalog migration, and it throws on every attempt — the consumer retries it endlessly, and everything queued behind it stops moving. A dead-letter policy sends any message that fails five times to a DLQ. There, an engineer sees a cluster of ≈40 failures all pointing at the same migration, patches the catalog data, and re-drives those 40 — which now succeed. A handful reference SKUs that are genuinely gone for good; those are discarded, each with a recorded reason. The main pipeline never stalled, and no order was silently dropped.

How it works¶

Threshold on failure. After a configured number of failed attempts (optionally with backoff first, to let transient faults recover), the item is routed aside rather than retried in place or deleted.
Preserve the item and its context. The dead letter carries the original payload plus failure metadata, so the fault can actually be diagnosed.
Triage in the side lane. A human or tool inspects, then dispositions: fix-and-replay, discard with a reason, or escalate — turning "it keeps failing" into a decision.

Tuning parameters¶

Failure threshold — how many attempts before dead-lettering. Low frees the main queue fast but dead-letters transient blips; high risks head-of-line blocking while a poison item is retried.
Retry/backoff before diversion — whether to retry with backoff first (transient network faults recover on their own) versus diverting immediately (deterministic faults never will).
Replay policy — automatic re-drive after a fix versus manual replay. Auto is fast but can silently re-poison the DLQ if the fix was wrong.
DLQ retention — how long dead letters live before they too must be dispositioned, so the side lane doesn't quietly become its own unbounded backlog.

When it helps, and when it misleads¶

Its strength is keeping a drain flowing past a few un-processable items while preserving them for diagnosis, rather than losing them to a silent drop or wedging the whole queue on an infinite retry.^[1]

Its failure mode is the write-only graveyard: a DLQ with no owner fills up unexamined, and the "handled" feeling is false — the obligations are deferred, not resolved. Auto-replay without fixing the root cause just cycles the same failures. The classic misuse is treating the DLQ as a dumping ground to make the main queue look drained while real work rots out of sight. The discipline is to give the dead-letter queue its own owner and disposition SLA — a dead letter is deferred, not done — and to treat a growing DLQ as an alarm, not a relief valve.

How it implements the components¶

disposition_path — the explicit fix-and-replay / discard / escalate outcomes for each diverted item.
exception_path — the failure-triggered diversion itself: the route that lets un-drainable items bypass the main order so they can't block it.

It does not expire items by age (TTL Expiration Sweep), inventory and triage a whole accumulated backlog (Incident Backlog Cleanup), or define when the overall drain is complete (Backlog Burn-Down).

Instantiates: Queue Draining — it keeps the main drain unblocked by routing un-processable items to a governed side lane.
Sibling mechanisms: TTL Expiration Sweep · Incident Backlog Cleanup · Message Queue Drain · Graceful Queue Shutdown · Connection Draining · Drain Dashboard · Surge Worker Pool · Backlog Burn-Down · Maintenance Drain · Appointment Waitlist Clearing

Notes¶

A dead-letter queue is itself a queue, and can accumulate into its own backlog that needs its own drain (often a Backlog Burn-Down or an Incident Backlog Cleanup). Diverting an item resolves nothing on its own — it only relocates the obligation to a place where it can be worked deliberately.

References¶

[1] A poison message is one that cannot be processed and, if retried in place, causes head-of-line blocking — it wedges every item behind it. Both are standard terms in message-queue design; the dead-letter queue is the standard remedy. ↩