Tensions in Practice: Independent progress in tension with duplicate rebuilding¶
Concurrent cache misses · one result key
Two requests find the same cache entry missing before either has rebuilt it. If both recompute independently, they can repeat identical source work. Coalescing the misses lets one request own the rebuild and the other wait for its result. That removes this duplicate computation but also makes the waiting request depend on the shared owner’s progress.
Keep requests independently serviceable
Let each miss obtain its answer without depending on another request’s in-flight state.
Avoid duplicate source work
Regenerate one shared result once when several requests need it together.
Why these aims pull against each other
Sharing an in-flight rebuild reduces work duplication by introducing ownership, joining and waiting. The result must be genuinely interchangeable for all joined requests.
Choose an arrangement to see what changes and what remains difficult.
Qualitative paths and conditions, not measured costs, timings or performance guarantees.
What this choice protects
What it costs
When it fits
Compare the arrangements
Rebuild independently
Let A and B each compute the safe, identical-key result and return it to their own caller.
- What it protects
- Each caller’s progress is independent of the other rebuild’s ownership record or failure.
- What it costs
- Concurrent misses duplicate source work and can amplify load when many arrive together.
- When it fits
- Fits cheap safe regeneration, rare concurrent misses and acceptable aggregate load where coordination would add little value.
Illustration note: Independent progress is an editorial consequence of the separate paths, not a claim that the requests lack shared infrastructure or cannot fail together.
Share one rebuild
Record one in-flight rebuild for the key; joining requests wait for its result rather than launching another copy.
- What it protects
- The illustrated concurrent pair generates one result instead of two copies.
- What it costs
- Callers wait on the owner; coordination, failure recovery and cancellation must be defined to avoid stranding them.
- When it fits
- Fits costly popular-key regeneration and requests whose authorization, inputs and freshness contract permit the same result to be shared.
Illustration note: The waiting/failure obligation is an explicit structural inference. The drawing shows success, not a complete timeout or owner-replacement protocol.
What this illustration does—and does not—establish
Caching: Cache Stampede / Thundering Herd on Miss supplies request coalescing as a response to concurrent misses. The two-request drawing exposes both reduced computation and the new shared-progress dependency.
- Freshness and cache validity are held fixed; coalescing does not establish either.
- The example uses safe result generation. Duplicating a side-effecting operation would require a separate correctness contract.
- Different users or inputs must not be merged merely because their requests look similar.
Source entries
Caching
Caching: Cache Stampede / Thundering Herd on Miss supplies the conflict examined here.
Cache Stampede / Thundering Herd on Miss
- T2: Cache Stampede / Thundering Herd on Miss. When a hot cached item expires or is invalidated, many concurrent requests may simultaneously attempt to regenerate or fetch it, overwhelming the source. Naive cache-aside patterns without locking or request coalescing can cause cascading failures under load when popular items expire; mitigation via probabilistic early refresh, request coalescing, or background refresh.
What It Is Not
- Not always beneficial. Caching imposes overhead (lookup, management, coherence) that must be offset by hit-rate savings. In workloads with poor locality (random access over a large dataset) or high coherence costs, caching can slow the system. "Cache-oblivious" or "cache-aware" algorithm design addresses when and how to cache.