Single-Flight Request Coalescing¶

Coordination protocol — instantiates Demand-Triggered Deferred Evaluation

When several callers demand the same not-yet-ready value at once, runs the computation a single time and hands that one result to every waiter.

Single-Flight Request Coalescing deduplicates demand across concurrency. When many callers ask for the same missing value in the same instant, the first becomes the leader and actually does the work; the rest register as waiters on that one in-flight computation and receive the leader's result when it lands. Its defining move is preventing the stampede — the moment where N callers all discover the same cache miss and all launch the same expensive fetch at once. This is distinct from memoization, which prevents recomputation over time by keeping a store; single-flight keeps no store at all, only a short-lived registry of what is currently in flight, and it is empty again the instant the leader finishes.

Example¶

A news site's homepage is served from a cache with a hot key that expires at the top of the minute. A breaking story sends traffic spiking, and at the instant the key expires, roughly 8,000 in-flight requests all miss simultaneously and — without coordination — all issue the same heavy database aggregation at once, a cache stampede that can topple the origin just as demand peaks.^[1]

With single-flight in front of the fetch, the first request to miss becomes the leader and runs the one aggregation. The other ~8,000 requests arriving during that window find an entry already in flight for that key, block on it, and are all served the leader's single result. The database sees one query instead of thousands, and every user still gets a fresh page. When the leader returns, the coalescing group dissolves; the next miss elects a fresh leader.

How it works¶

Coalescing key. An in-flight registry keyed by request identity. Callers whose key matches an entry already in flight attach as waiters instead of starting their own run.
Leader election and fan-out. Exactly one caller per key executes; on completion its result is delivered to the leader and every attached waiter under a single result-sharing policy.
Abandonment handling. Define what happens if the leader fails, times out, or is cancelled: either propagate the error to all waiters, or promote a waiter to re-run — never silently hand out a half-formed result.
In-flight only. The registry holds current work, not past results; persistence is deliberately somebody else's job.

Tuning parameters¶

Coalescing key breadth — what counts as "the same request." Too coarse and you merge callers who differ in an authorization or parameter that changes the answer; too fine and you never coalesce anything.
Leader-failure policy — fail all waiters vs. re-elect and retry. Re-election is more resilient but can amplify load against a struggling dependency.
Waiter timeout — how long a waiter blocks on the leader before giving up. Guards against one slow leader stalling every attached caller (head-of-line blocking).
Scope — per-process vs. cross-node coalescing (via a distributed lock or lease). Wider scope suppresses more duplication but adds coordination latency and failure modes.

When it helps, and when it misleads¶

Single-flight shines at cache-miss stampedes and any expensive, idempotent fetch that many clients want at once — it caps concurrent duplication at one regardless of how sharp the demand spike is, and it is the natural front for a cache it fills exactly once.

Its failure modes come from the two things it centralizes: the shared key and the shared leader. The dangerous misuse is a coalescing key too broad — folding requests that differ in a security-relevant parameter (a user id, a tenant, a scope) onto one key, so one caller's result is served to another. And because every waiter's fate is tied to the leader, a slow or dying leader blocks or fails the whole group at once. The discipline: make the key total over every field that affects the result, including authorization; bound the waiter timeout; and treat leader abandonment as a first-class case — re-elect or fail cleanly, never publish a partial result as if it were whole.

How it implements the components¶

single_flight_and_duplicate_suppression_rule — the coalescing rule itself: at most one concurrent execution per key, all other demanders attach as waiters.
result_sharing_policy — the fan-out that delivers the leader's single result to every waiter in the group.
cancellation_and_abandonment_rule — the policy for a leader that fails, times out, or is cancelled: propagate or re-elect, never leak a half-result.

It does NOT persist results for future demands (memoized_result_store, invalidation_rule — that's Memoization) and it does NOT decide *whether the value is needed at all (need_predicate — that's the Conditional Workflow Gate); it assumes the value is wanted now and only collapses the concurrent duplicates.*

Instantiates: Demand-Triggered Deferred Evaluation — single-flight is the concurrency-side guard that keeps deferred work from being realized many times at once.
Consumes: Memoization — single-flight typically sits in front of a cache and populates it with the leader's one result.
Sibling mechanisms: Memoization · Thunk or Lazy Promise · Demand-Driven Build Graph · On-Demand Report Generation · Deferred Database Query Object

Notes¶

The leader-abandonment case is the subtle one and worth deciding up front rather than discovering in an incident: if the leader is cancelled or its context is torn down, a naïve implementation can deliver the cancellation to every waiter, converting one caller's disconnect into a group-wide failure. Choosing between "fail all" and "promote a new leader" is a real design fork, not an implementation detail.

References¶

[1] A cache stampede (or thundering herd) is the load spike that occurs when a popular cached value expires and many concurrent requests all miss and recompute it at the same instant. Request coalescing is one of the standard mitigations, alongside probabilistic early expiration and serving stale-while-revalidate. ↩