Skip to content

Consumer Group

Load-sharing consumer pool — instantiates Topic-Brokered Event Distribution

A pool of cooperating consumers that split one subscription's event stream across partitions, so throughput scales with instances while each event is handled once within the group.

A Consumer Group is a set of consumer instances that share a single logical subscription rather than each receiving the whole of it. The stream is divided into partitions, each partition is assigned to exactly one member, and the group collectively processes the topic once — so adding instances adds throughput. Its distinguishing move is that it inverts fan-out within the subscriber: where the broker copies an event to every subscription, a consumer group ensures an event goes to exactly one member of the group. This is the load-sharing, competing-consumers side of pub/sub — the mechanism that lets a subscription that would overwhelm one process be handled by twenty, and that absorbs bursts by spreading them, without any single event being processed twice inside the group.

Example

An ad-tech platform ingests a clickstream on events.impressions at a volume no single process could keep up with. The impression-aggregation service runs as a consumer group of twelve instances. The stream is split into forty-eight partitions, and each instance is assigned four; every impression lands in exactly one partition and is therefore processed by exactly one instance. Throughput is roughly twelve times a single consumer's.

When a traffic spike arrives, operators start eight more instances. The group rebalances: partitions are reassigned so the twenty members each hold a smaller share, and the backlog drains faster. When the spike passes and instances are removed, their partitions are redistributed to the survivors. One slow instance only backs up its own partitions, so its lag is isolated rather than stalling the whole subscription — and the group keeps the guarantee that each impression is aggregated once.

How it works

  • Partition the stream. The subscription's events are divided into partitions (typically by a key), the unit of parallelism and of ordering.
  • Assign one owner per partition. Each partition is held by exactly one group member at a time, which is what makes "processed once within the group" true while still running in parallel.
  • Rebalance on membership change. When instances join or leave, partitions are reassigned across the current members so load stays roughly even.
  • Isolate lag to a partition. A slow or stuck member delays only the partitions it owns; the rest of the group proceeds, containing the blast radius of one bad consumer.

Tuning parameters

  • Partition count — the ceiling on parallelism. More partitions allow more instances but add coordination overhead and can fragment per-key ordering.
  • Partitioning key — what determines which partition an event lands in. A good key spreads load evenly and keeps related events together; a skewed key creates hot partitions.
  • Rebalance strategy — stop-the-world reassignment versus incremental/cooperative rebalancing. Incremental disrupts fewer members but is more complex.
  • Instance count vs. partitions — more instances than partitions leaves some idle; fewer means each owns several. This dial sets headroom and utilization.
  • Max in-flight per member — how many events a member processes before committing progress. Higher throughput, but more redelivered work if that member dies mid-batch.

When it helps, and when it misleads

Its strength is horizontal scale with a once-per-group guarantee: a subscription becomes as fast as its instance count, bursts are absorbed by spreading them, and a single failing consumer is contained to its partitions rather than halting the stream — the competing-consumers pattern[1] applied inside one subscriber.

It misleads when partitioning is careless. A skewed key produces hot partitions that no amount of extra instances relieves, because the bottleneck is one partition owned by one member. Adding instances beyond the partition count buys nothing — surplus consumers sit idle — which teams routinely misdiagnose as a throughput ceiling in the broker. And because ordering holds only within a partition, code that assumes global ordering across the group will see events "out of order" that were never promised in order. The discipline is to choose a partition key that spreads load and preserves the ordering you actually depend on, size partitions ahead of the parallelism you'll need, and treat cross-partition ordering as something you design for, not assume.

How it implements the components

  • consumer_group_partition_plan — it defines the partitioning of the stream and the assignment of partitions to members: the plan that turns one subscription into parallel, once-each processing.
  • backpressure_and_isolation_profile — by spreading load across instances and confining a slow member's lag to its own partitions, it absorbs bursts and isolates a struggling consumer from the rest of the group.

It does not route or fan events out to subscriptions in the first place (Fan-Out Exchange), persist a backlog for an offline subscriber (Durable Subscription Queue), run the per-event acknowledgement protocol (Delivery Acknowledgement), or protect *other subscriptions from a slow consumer (Slow Consumer Isolation).*

References

[1] The competing consumers pattern — several consumers drawing from the same subscription so that each message is handled by exactly one of them, used to scale throughput and share load. It is the load-balancing counterpart to publish-subscribe fan-out, applied inside a single subscription. ↩