Skip to content

Slow Consumer Isolation

Backpressure and isolation policy — instantiates Topic-Brokered Event Distribution

Contains a slow or stuck subscriber so its backlog can't stall the broker or starve healthy consumers, keeping one lagging handler from becoming everyone's outage.

Slow Consumer Isolation draws a wall around each subscriber so that one that cannot keep up degrades alone. In a broker fanning one topic out to many consumers, a single handler that stalls — a downstream database locked, a service timing out — will, left unchecked, back up shared buffers until every other consumer on that topic slows too. Its defining move is to make backpressure selective: instead of slowing producers for everyone (global backpressure) or dropping the topic, it bounds and isolates the offender's share of resources, so the blast radius of a slow consumer is that consumer. It reads each subscriber's declared consumption profile to know what "keeping up" means, and enforces the consequence when it doesn't.

Example

A live-scores broker fans match events out to hundreds of thousands of mobile clients and, alongside them, a single archival consumer that writes every score change to cold storage. During a marquee match the archival consumer's storage backend slows, and it falls behind. Without isolation, its unread backlog piles up in shared broker memory until live mobile consumers start lagging too — and fans see stale scores during the one moment they are watching. With isolation, the archival consumer has its own bounded buffer and quota; when it overflows, only that consumer is throttled or spilled to its own overflow store, while the live fan-out stays at the head of the stream. The slow consumer stays slow; everyone else stays current.

How it works

  • Per-consumer bounded buffers — each subscriber gets its own capacity, not a share of one common pool, so a full buffer belongs to one consumer.
  • A defined overflow action — when a consumer's buffer fills, an explicit policy fires for that consumer only: throttle it, drop-oldest, or spill to a private overflow queue.
  • Resource bulkheads — connections, threads, and memory are partitioned so exhaustion in one lane cannot drain another[1].
  • Lag as the trigger — the isolation profile watches each consumer's distance behind the live edge against its handler contract, and acts on the laggard rather than the topic.

Tuning parameters

  • Buffer depth per consumer — how much a subscriber may fall behind before action; deeper tolerates bursts but ties up more memory per consumer.
  • Overflow action — drop-oldest (favor freshness, accept loss), pause (favor completeness, accept lag), or spill-to-durable (favor completeness, pay storage); pick by whether that consumer can tolerate gaps.
  • Isolation granularity — isolate per subscription, per consumer instance, or per consumer group; finer isolation is safer but multiplies bookkeeping.
  • Throttle-versus-evict threshold — how far behind before a consumer is merely slowed versus cut loose and marked unhealthy.
  • Fairness quota — the ceiling on any one consumer's share of broker resources, protecting the many from the greedy few.

When it helps, and when it misleads

Its strength is eliminating cross-subscriber head-of-line blocking: one slow consumer can no longer take a whole topic down with it, and the broker degrades gracefully under an uneven load. Its failure mode is silent isolation — a consumer quietly having events dropped or paused while believing it is caught up, so the containment hides a real data loss. A related misuse is enlarging buffers indefinitely to avoid ever dropping, until the broker itself runs out of memory and the isolation that was meant to protect it becomes the outage. The discipline is to make every isolation event loud: pair it with the Subscription Health Dashboard and alerting, so an isolated consumer is treated as a signal to fix or reprovision it, not a state to leave running forever.

How it implements the components

  • backpressure_and_isolation_profile — it is the profile: the per-consumer capacity limits, overflow actions, and resource bulkheads that keep one subscriber's slowness local.
  • subscriber_handler_contract — it reads and enforces each handler's declared throughput and acknowledgement behavior to decide what counts as falling behind and when to act.

It contains a slow consumer but does not park individual messages that repeatedly fail (that's Dead-Letter Queue), hold a durable backlog for a subscriber that is entirely offline (that's Durable Subscription Queue), or surface the lag it reacts to (delivery_trace_record — Subscription Health Dashboard).

References

[1] The bulkhead pattern — named for the sealed compartments that keep a breach in one part of a ship's hull from flooding the rest — partitions resources so that failure or overload in one component cannot exhaust the resources another depends on. Slow Consumer Isolation is the bulkhead applied to a broker's consumers. ↩