Skip to content

Durable Subscription Queue

Durable subscriber buffer — instantiates Topic-Brokered Event Distribution

A per-subscriber queue that persists unacknowledged events across disconnects and restarts, so a consumer that was offline still receives everything it missed.

A Durable Subscription Queue gives a subscriber a private, persistent backlog: events matching its subscription are stored for that subscriber and held until it has consumed them, even if the subscriber is disconnected, restarting, or slow. Its distinguishing property is that the subscription outlives the connection — the broker remembers this subscriber's place and keeps its unconsumed events on disk, so downtime becomes lag to catch up on rather than data lost. This is what separates it from best-effort delivery, where an event fanned out to an absent consumer simply evaporates. The queue is bound to the identity of the subscription, not to a live socket, and the retention policy on it decides how long the system is willing to wait for that subscriber to come back.

Example

A billing-reconciliation service consumes invoices.issued to keep the ledger in step. It goes down for a two-hour maintenance window. During that window the invoicing system keeps publishing — dozens of events the reconciler is not there to receive.

Because the reconciler holds a durable subscription, the broker does not discard those events. It writes each one to the reconciler's queue and marks the subscriber's position. When the service comes back, it finds its backlog intact and processes the two hours of invoices in order, from where it left off, then catches up to live. No invoice is missed and none has to be manually re-fetched. A best-effort subscriber would have silently lost the window; the durable queue converts an outage into a bounded catch-up governed by how long the retention policy keeps the backlog.

How it works

  • Per-subscriber persistence. Each durable subscription gets its own stored queue; matched events are written to it and survive broker restarts.
  • Position tracking. The broker records how far this subscriber has consumed, so delivery resumes exactly at the gap rather than from the start or from "now."
  • Retention governs the wait. A policy caps how long or how large the backlog may grow before old events are aged out — the bound on how patient the system is with an absent consumer.
  • Lifecycle beyond the connection. The subscription is created, persists across disconnects, and is explicitly torn down; while it exists the broker keeps its backlog whether or not the consumer is attached.

Tuning parameters

  • Retention horizon — how long unconsumed events are kept (time or size bound). Longer tolerates longer outages but costs storage and can hide a permanently-dead consumer.
  • Durability level — in-memory, single-disk, or replicated persistence. Stronger durability survives worse failures at higher latency and cost.
  • Overflow policy — on a full backlog, block the publisher, drop oldest, or drop newest. This dial decides who suffers when a consumer falls too far behind.
  • Subscription teardown — whether an idle durable subscription is auto-expired after some period or must be explicitly deleted. Auto-expiry reclaims storage but can silently drop a slow returner.
  • Delivery resumption — replay strictly from the last position versus jump to latest on reconnect. Strict replay loses nothing; jumping trades completeness for freshness.

When it helps, and when it misleads

Its strength is turning consumer availability from a correctness problem into a capacity problem: a subscriber may crash, deploy, or lag, and still lose nothing, because its place and its backlog are the broker's responsibility, not the network's. This durable-subscription guarantee[1] is what lets otherwise-unreliable consumers participate safely in the stream.

It misleads when retention is treated as free or infinite. A durable subscription for a consumer that never returns becomes an unbounded, invisible cost — a queue growing forever behind a dead service — and a too-short retention silently drops events during a longer-than-expected outage, reintroducing the loss it was meant to prevent right when things are already going wrong. It is also easy to confuse with a replayable event log: the durable queue serves this subscriber's forward progress, not arbitrary time-travel over history. The discipline is to bound retention deliberately, alarm on backlogs that only grow, and reach for a replay log when the need is re-reading history rather than catching an offline consumer up.

How it implements the components

  • replay_and_retention_policy — it persists each subscriber's unconsumed events and enforces how long they are kept, so a returning consumer can be brought current from where it stopped.
  • subscription_lifecycle_rule — the durable subscription is one whose existence spans disconnects: created, retained through downtime, and explicitly expired, with the queue's fate tied to that lifecycle.

It does not confirm each event was processed (Delivery Acknowledgement), decide where permanently-failing events go (Dead-Letter Queue), route or fan events out (Message Broker, Fan-Out Exchange), or serve open-ended historical replay from a shared log (Replay Log or Event Stream).

References

[1] A durable subscription (as in the JMS sense) is one the broker retains even while the subscriber is disconnected, holding its matched messages until it returns — as opposed to a non-durable subscription that receives only what is published while it is connected. ↩