# B-vs-B' grade-set: S2_service_reliability — Cascading failure in a high-traffic online service

SCENARIO (given verbatim):
A high-traffic online service is suffering intermittent total outages. A modest spike in user requests causes a few backend nodes to slow down; clients and internal callers respond by automatically retrying failed requests, which multiplies load on the already-struggling nodes, which pushes more nodes over the edge, until the whole service is saturated and unavailable. Recovery only happens when operators manually shed nearly all traffic and bring the system back slowly. The team has good monitoring and can change client and server behavior, set limits, and add capacity, but cannot indefinitely over-provision hardware.

Decision required: Design the mechanisms that keep a local slowdown from escalating into a full outage, and that let the system recover on its own.

Success criteria: A localized slowdown stays localized (no full-service saturation) AND the system returns to normal service without manual traffic-shedding within minutes. Failure = a single hot spot can still cascade to total outage, OR the safeguards throttle so aggressively that normal traffic is routinely degraded.

4 candidate recommendations, labeled R1, R2, R3, R4.

## Recommendation R1
The outage is not a capacity problem; it is an amplification problem. A small slowdown is harmless on its own — what makes it total is that every failed or slow request is answered with more requests, so the system's own response to trouble becomes the fuel for more trouble. Each retry is individually reasonable but collectively destructive: a caller that retries gets a marginally better chance for itself while adding load that everyone, including itself, pays for. The whole design below has one goal: make the load added per unit of trouble less than one, so disturbances shrink instead of growing, and build a recovery path that the system walks by itself.

**Cap the amplifier at the source (retries).** Give every client and internal caller a *retry budget*: retries may be at most a small fraction (start at 10%) of that caller's recent successful requests, enforced over a rolling window. When the budget is exhausted, failures are surfaced immediately rather than retried. Retries must use exponential backoff with full random jitter (so callers do not re-converge into synchronized waves) and must stop entirely once a response carries an explicit "overloaded, do not retry" signal. Forbid nested retries: a request that is itself a retry of an upstream call is never retried again, so a single user action cannot fan out into a storm.

**Make overloaded servers reject fast instead of dying slow.** Each node advertises load via response headers. When a node's queue depth or latency crosses a high-water mark, it sheds *new* load by rejecting requests quickly with a "retry-after" hint and the no-retry signal, while protecting in-flight work — a fast cheap rejection cannot itself cascade, but a slow timeout can. Shed lowest-priority traffic first (background, best-effort, then bulk) so essential traffic keeps flowing; this is what keeps normal service from being broadly degraded.

**Cut the coupling so a hot spot stays a hot spot.** Partition callers and dependencies into isolated pools (bulkheads) with separate connection/concurrency limits, so one saturated dependency cannot consume the threads or connections that healthy paths need. Put a *circuit breaker* in front of each dependency: when a callee's error or timeout rate over a short window exceeds a threshold (e.g. 50% over 10 seconds), the breaker opens and calls fail fast locally for a cooldown, removing load from the struggling node entirely. The breaker is the firebreak — it stops re-emission rather than merely slowing it.

**Recover automatically, with a deliberate gap between tripping and re-arming.** Recovery is not the reverse of the failure; it is a separately engineered ramp. After cooldown the breaker goes *half-open* and admits a trickle of probe traffic; it closes only after sustained success, and re-opens on the first renewed failures. Use *different thresholds to shed and to recover* than the levels at which you stop shedding — recover at a comfortably lower load than the level that triggered shedding (e.g. trip at 80% utilization, resume at 60%). This gap prevents the system from flapping on and off at the boundary. When a node recovers, ramp its admitted traffic back up in stages (e.g. 10% → 25% → 50% → 100%), advancing only while latency and error rates stay healthy and stepping back automatically if they degrade, so a node that is not truly well is not slammed back to full load.

**Keep it from quietly over-protecting.** Add a deadband so ordinary variation triggers nothing, and treat *frequent or sustained* shedding, open breakers, or exhausted retry budgets as alarms — not as normal operation — because that signature means real capacity is short and the safeguards are now masking it. Add capacity or raise limits in response to that signal, not to transient spikes.

**Operating rule:** every component, at every moment, must add less load in response to trouble than the trouble it is responding to — cap retries, reject fast, isolate and break the coupling at defined load thresholds, and re-admit traffic only gradually and only once health is sustained, with a clear gap between the level that triggers protection and the lower level that releases it. If a localized slowdown can still recruit the rest of the system into amplifying it, the loop is not yet closed.

## Recommendation R2
The outages are a self-feeding loop: a few slow nodes cause timeouts, timeouts trigger automatic retries, retries multiply load on exactly the nodes that are already struggling, and the overload spreads until everything is saturated. Because the level of load that ignites the loop is far higher than the level at which it dies down on its own, the system cannot climb back out without a human throwing away nearly all traffic. The design therefore has two jobs that must be built together: starve the amplifying loop at the point where it regenerates, and give the system a staged, self-driven way back.

Start at the retries, because that is where a small fault becomes a flood. Replace blind immediate retries with capped retries that back off and add randomness: at most one or two retries per request, each after an exponentially growing, jittered delay so callers do not resynchronize into waves. Govern retries with a budget — a caller may spend retries on no more than a small fraction (for example 10%) of its recent requests; once that budget is exhausted it stops retrying and fails fast. A failed-fast call should return a cached or default answer where one exists, so that cutting the retry does not simply convert load into errors. This single change removes most of the multiplier.

Wrap every call to a backend or dependency in an automatic trip. Each caller continuously watches the error rate and latency of each dependency; when either crosses a threshold, the caller stops sending new requests to that dependency and immediately serves fallbacks instead of piling up on timeouts. Keep the trip scope narrow — per dependency and, where possible, per shard or partition — so one hot partition is isolated without removing healthy capacity. Crucially, the threshold to re-engage must be stricter than the threshold that tripped it, and re-engagement must be a single trial request (a probe) that is promoted only if it succeeds: this gap prevents the breaker from oscillating rapidly between open and closed, which is worse than either state.

Protect the servers from the inside too. Give each node a bounded concurrency limit and a bounded queue; when the queue is full, reject new work at the door rather than accepting it and slowing everything already in flight. Shed by priority: health checks, critical user paths, and internal control traffic get reserved capacity and are admitted first; bulk, low-priority, and best-effort traffic are the first to be rejected when the node is near its limit. Partition callers and workloads into separate pools (bulkheads) so a flood in one workload cannot consume the threads, connections, or memory that another depends on. The combination of bounded queues, concurrency caps, and priority admission ensures a node degrades gracefully — staying responsive to important work — instead of collapsing.

These guards must not strangle normal traffic, which is the opposite failure. Size every limit from observed normal peaks plus headroom, exempt health checks and critical paths from shedding, and make every trip, shed, and rejection visible in monitoring with its reason, so a too-tight limit is caught and retuned rather than silently degrading service.

Recovery should be automatic and gradual, never a manual cliff. When a node or dependency recovers, do not return it to full traffic at once. Ramp it: admit a small share of traffic, watch error rate and latency for a short interval, and only increase the share if the metrics hold; if they degrade, step back down automatically. Pair this with autoscaling that adds capacity when sustained healthy load approaches limits — within a ceiling, since hardware cannot grow without bound — so the staged ramp has somewhere to grow into. The ramp rate must stay below the measured spare headroom, or the act of returning traffic will reignite the very loop you just broke.

The final operating rule: never let a failed call generate more total load than the original call, and never return a recovering component to full load faster than it can be proven to absorb. Cut the retry multiplier at its source, isolate faults narrowly with hysteretic trips, shed and queue by priority so nodes degrade instead of dying, and bring everything back on a monitored ramp that backs off at the first sign of relapse. A local slowdown then stays a local slowdown, and the system climbs back on its own within minutes.

## Recommendation R3
The outages are not a capacity problem; they are a feedback problem. A small slowdown is being amplified by the very thing meant to cope with it — automatic retries — turning a local dip into a self-reinforcing loop that spreads across nodes until everything saturates. The fix is to cut the amplifier, give every overloaded part a way to protect itself, and build in an automatic, gradual return to normal so no human has to shed traffic.

**Cut the amplifier.** Retries are the engine of collapse, so make them cheap and self-limiting. Cap retries to at most one attempt, and only retry errors that are actually retryable. Gate every retry behind a token-bucket retry budget per client: if more than roughly 10% of recent requests are already retries, stop retrying and fail fast. Add full jitter to all backoff so that callers who failed together do not return in a synchronized wave. Critically, a node returning an overload signal must tell callers *not* to retry it (a "retry-after / do-not-retry" response), converting a retry storm into immediate, quiet shedding.

**Let each node protect itself.** Give every server an admission controller that watches its own live latency and queue depth, not just CPU. When in-flight concurrency or queue latency crosses a threshold, the node immediately rejects excess work with a fast, cheap overload response rather than slowly grinding through it — shedding load is what keeps it from tipping. Bound every queue; an unbounded queue just hides the collapse. Shed by priority class: health checks and high-value user traffic are admitted last to be dropped, bulk and background work first, so protection never degrades normal interactive traffic until genuinely necessary.

**Stop failure from spreading.** Put a circuit breaker on every call between services and to shared dependencies. When error or latency from a downstream target crosses a threshold, the breaker opens and callers fail fast (serving degraded or cached responses) instead of piling load onto a struggling target. This isolates a hot spot so it cannot drag its callers down with it. Apply concurrency limits per downstream dependency so one slow dependency cannot consume all of a caller's worker threads.

**Make recovery automatic and gradual.** Recovery must not be a single switch back to full load — that simply re-creates the overload that caused the outage. After a breaker opens, it moves to a half-open state and admits only a trickle of probe traffic. Real traffic is then ramped back in stages (for example, 1% → 10% → 50% → 100%), and at each stage the controller watches the same latency and error signals that triggered the trip. If they stay healthy for a short hold period, it advances; if they degrade, it steps back down. Jitter and stagger these ramp-ups across nodes and clients so recovery itself does not arrive as a synchronized surge. This is the balancing loop the system currently lacks: it senses load, corrects, and walks itself back to normal within minutes without an operator.

**Keep the safeguards from over-correcting.** The danger is throttling so hard that normal traffic is routinely degraded. Prevent this by keying limits to live measured latency rather than fixed caps, so headroom expands automatically when the system is healthy; by using hysteresis (trip quickly, recover only after a sustained healthy interval) so the system does not flap between open and closed; and by tuning thresholds so that under normal load the breakers stay closed and admission controllers reject essentially nothing. Use any spare capacity and fast autoscaling as a buffer for genuine spikes, but treat shedding and breaking — not over-provisioning — as the primary defense, since hardware cannot be scaled without limit.

**Final operating rule.** Each component continuously measures its own load and the health of everything it depends on. When load exceeds what it can serve within target latency, it sheds or fails fast immediately — lowest-priority work first — rather than absorbing the overload; when a dependency is unhealthy, it stops sending it work. Recovery is never a single jump back to full load: traffic is re-admitted in jittered stages, advancing only while the health signals stay green and stepping back the moment they do not. A local slowdown thus stays local, and the system returns itself to full service without anyone manually shedding traffic.

## Recommendation R4
The outages are not caused by the demand spike itself but by how the system reacts to it: a failed or slow response triggers automatic retries, retries pile new load onto the nodes that are already struggling, and that extra load pushes more nodes over the edge. The fix is to cut the strength of that self-feeding loop, stop a local failure from spreading, and give the system a built-in way to settle back to normal on its own.

Start at the retries, because that is where small problems get multiplied. Every client and internal caller should retry only failures that are genuinely worth retrying, cap retries at one or two attempts, and space attempts using exponential backoff with random jitter so callers do not resynchronize into a fresh wave. Most importantly, give each caller a retry budget: allow retries to consume at most a small fraction (for example 10%) of its request volume over a rolling window, and when that budget is exhausted, stop retrying and fail fast. A retry budget is what structurally prevents a retry storm, because total load can no longer balloon to a multiple of real demand.

Next, keep a hot spot from becoming a system-wide fire by adding boundaries that the failure cannot cross. Give each downstream dependency its own bounded pool of concurrent calls (a bulkhead) so that one slow dependency cannot consume all the threads, connections, or memory and starve unrelated work. In front of each dependency or node, run a circuit breaker: track its recent error and latency rate, and when that crosses a trip threshold, open the breaker and immediately reject calls to it for a short cool-off instead of queuing them. Rejecting cheaply and instantly is what protects the struggling node and lets it drain its backlog and recover.

Make the system shed the right load rather than collapse under all of it. Put a bounded queue with admission control in front of each node; when the queue is near full or latency climbs, reject the lowest-priority excess at the edge before it consumes capacity, returning a clear "busy, retry later" signal (which, combined with the retry budget, will not rebound into a storm). Classify traffic by importance so that under pressure the system preserves critical requests and sheds optional ones first. This is also the safeguard against the opposite failure: because shedding is prioritized and sized to protect spare capacity, normal traffic is not routinely degraded — only genuine overload is touched.

To remove the need for manual recovery, close a self-correcting loop on each node. Continuously measure each node's latency and error rate against a healthy target band, and let an adaptive concurrency limiter raise the allowed in-flight requests when the node is healthy and lower it when latency or errors rise — an automatic, fast-acting throttle that finds the safe load level on its own. Use asymmetric thresholds (hysteresis): trip protections quickly when health degrades, but require sustained good health before fully re-admitting traffic, so the system does not flap between open and closed. The circuit breaker enforces this directly: after its cool-off it goes half-open, lets a trickle of probe requests through, and only fully reopens if those succeed, then ramps traffic back gradually rather than all at once. Pair this with autoscaling as a slower loop that adds capacity when sustained healthy demand is high, while the fast loops hold the line in the seconds before new capacity arrives.

Finally, prefer degrading over failing. When a dependency's breaker is open, return a cached, partial, or default response where the request allows it, so a local outage shows up as reduced richness rather than a hard error that provokes more retries. Make write operations idempotent (with request keys) so the bounded retries that remain are safe. Keep monitoring on the new safeguards themselves — shed rate, breaker open/close events, retry-budget exhaustion, and the adaptive limits — so you can see both kinds of trouble: a cascade leaking through, or throttling that is biting healthy traffic.

Operating rule: every caller retries sparingly within a strict retry budget; every dependency runs behind a bulkhead and a circuit breaker that trips fast and re-admits slowly via half-open probing; each node self-tunes its accepted load to a healthy latency/error band and sheds prioritized excess early; capacity scales on the slow loop. A slowdown then stays contained to the affected component, and the system walks itself back to full service in minutes — with manual traffic-shedding kept only as a never-needed last resort.
