# Grade-set: S2_service_reliability — Cascading failure in a high-traffic online service

SCENARIO (given verbatim to both reasoning methods):
A high-traffic online service is suffering intermittent total outages. A modest spike in user requests causes a few backend nodes to slow down; clients and internal callers respond by automatically retrying failed requests, which multiplies load on the already-struggling nodes, which pushes more nodes over the edge, until the whole service is saturated and unavailable. Recovery only happens when operators manually shed nearly all traffic and bring the system back slowly. The team has good monitoring and can change client and server behavior, set limits, and add capacity, but cannot indefinitely over-provision hardware.

Decision required: Design the mechanisms that keep a local slowdown from escalating into a full outage, and that let the system recover on its own.

Success criteria: A localized slowdown stays localized (no full-service saturation) AND the system returns to normal service without manual traffic-shedding within minutes. Failure = a single hot spot can still cascade to total outage, OR the safeguards throttle so aggressively that normal traffic is routinely degraded.

There are 6 candidate recommendations below, labeled R1, R2, R3, R4, R5, R6.

## Recommendation R1
The outages are not really caused by the demand spike; they are caused by the system's own reaction to the spike. When a few nodes slow down, callers retry, and those retries pile fresh load onto the exact nodes that are already struggling, which slows them further and recruits their neighbors. The fix has to attack that self-feeding loop at three points: stop callers from amplifying trouble, stop a sick node from poisoning the healthy ones, and let the system climb back on its own.

Start at the caller, because that is where the amplification lives. Replace unconditional retries with disciplined ones: at most one or two retries, each after an exponentially growing delay with random jitter so callers do not resynchronize into waves. Cap retries with a budget — a caller may spend retries on no more than roughly five to ten percent of its requests in any window; once the budget is exhausted it fails fast instead of hammering. Make retries respect server backpressure: when a node is overloaded it returns an explicit "busy, do not retry yet" signal with a retry-after hint, and callers obey it. Only idempotent operations may be retried, and those carry idempotency keys so a retry never duplicates work.

Around every call to a backend dependency, wrap a circuit breaker. It watches that dependency's error rate and tail latency; when either crosses a threshold the breaker opens and calls fail immediately, returning a cached or default response rather than queuing behind a dying node. Crucially, scope each breaker to a single dependency and give each dependency its own bounded concurrency pool (a bulkhead) with a small fixed number of in-flight slots and bounded, deadline-aware queues. A slow node can then fill only its own pool; it cannot consume the caller's entire thread or connection budget and drag everything down. Requests that cannot get a slot are rejected quickly with a clear error — never silently dropped, never parked in an unbounded queue that hides a growing backlog.

When overall pressure still rises, shed load by priority rather than at random. Tag traffic into classes: essential user-facing reads and writes are protected; speculative, batch, and low-value internal calls are sacrificed first. Trigger shedding from an early-warning threshold that preserves headroom — act before saturation, not at the cliff — and use separate, wider thresholds for starting and stopping shedding (a hysteresis band) so the system does not flap on and off near the line. Shedding must be fair within a class so no single tenant or region is starved, and it must always leave the protected class genuinely served; if it cannot, that is the signal to add capacity, not to shed deeper.

Recovery must be a designed, staged process, not a hope. Exit from trouble is fast and blunt; return is slow and gated, because the moment of re-engagement is when the system is most fragile and a premature reconnect simply re-lights the fire. After a breaker has been open for a cooldown, let exactly one trial request through. If it succeeds, restore traffic in stages — a small percentage first, then more — advancing only while error rate and latency stay healthy, and dropping back a stage the instant they regress. Tie this to bounded, automatic capacity scaling: add nodes in response to sustained healthy load to meet real demand, but never as a substitute for governing demand, since fresh capacity behind ungoverned retries is consumed instantly.

Tune so that normal traffic is never routinely degraded: thresholds sit above ordinary peak behavior with margin, breakers trip on genuine distress rather than noise, and shedding touches only sacrificeable classes until protected load is actually at risk. Exercise the whole arrangement with regular fault-injection drills — deliberately make a node hot and confirm that the slowdown stays contained, that protected traffic keeps flowing, and that the system ramps back by itself within minutes. Keep a manual drain switch as a last-resort escalation only; if operators ever have to use it, treat that as a defect in the automatic loop.

Final operating rule: every caller throttles itself under failure instead of amplifying it; every dependency is isolated behind its own breaker and bounded pool so a local fault stays local; under pressure the system sheds by priority with headroom and hysteresis rather than collapsing; and it returns to full service through a staged, self-driven, automatically reversible ramp — so a hot spot degrades a slice, never the whole, and heals without a human pulling traffic.

## Recommendation R2
The outages are not caused by the demand spike itself but by how the system reacts to it: a failed or slow response triggers automatic retries, retries pile new load onto the nodes that are already struggling, and that extra load pushes more nodes over the edge. The fix is to cut the strength of that self-feeding loop, stop a local failure from spreading, and give the system a built-in way to settle back to normal on its own.

Start at the retries, because that is where small problems get multiplied. Every client and internal caller should retry only failures that are genuinely worth retrying, cap retries at one or two attempts, and space attempts using exponential backoff with random jitter so callers do not resynchronize into a fresh wave. Most importantly, give each caller a retry budget: allow retries to consume at most a small fraction (for example 10%) of its request volume over a rolling window, and when that budget is exhausted, stop retrying and fail fast. A retry budget is what structurally prevents a retry storm, because total load can no longer balloon to a multiple of real demand.

Next, keep a hot spot from becoming a system-wide fire by adding boundaries that the failure cannot cross. Give each downstream dependency its own bounded pool of concurrent calls (a bulkhead) so that one slow dependency cannot consume all the threads, connections, or memory and starve unrelated work. In front of each dependency or node, run a circuit breaker: track its recent error and latency rate, and when that crosses a trip threshold, open the breaker and immediately reject calls to it for a short cool-off instead of queuing them. Rejecting cheaply and instantly is what protects the struggling node and lets it drain its backlog and recover.

Make the system shed the right load rather than collapse under all of it. Put a bounded queue with admission control in front of each node; when the queue is near full or latency climbs, reject the lowest-priority excess at the edge before it consumes capacity, returning a clear "busy, retry later" signal (which, combined with the retry budget, will not rebound into a storm). Classify traffic by importance so that under pressure the system preserves critical requests and sheds optional ones first. This is also the safeguard against the opposite failure: because shedding is prioritized and sized to protect spare capacity, normal traffic is not routinely degraded — only genuine overload is touched.

To remove the need for manual recovery, close a self-correcting loop on each node. Continuously measure each node's latency and error rate against a healthy target band, and let an adaptive concurrency limiter raise the allowed in-flight requests when the node is healthy and lower it when latency or errors rise — an automatic, fast-acting throttle that finds the safe load level on its own. Use asymmetric thresholds (hysteresis): trip protections quickly when health degrades, but require sustained good health before fully re-admitting traffic, so the system does not flap between open and closed. The circuit breaker enforces this directly: after its cool-off it goes half-open, lets a trickle of probe requests through, and only fully reopens if those succeed, then ramps traffic back gradually rather than all at once. Pair this with autoscaling as a slower loop that adds capacity when sustained healthy demand is high, while the fast loops hold the line in the seconds before new capacity arrives.

Finally, prefer degrading over failing. When a dependency's breaker is open, return a cached, partial, or default response where the request allows it, so a local outage shows up as reduced richness rather than a hard error that provokes more retries. Make write operations idempotent (with request keys) so the bounded retries that remain are safe. Keep monitoring on the new safeguards themselves — shed rate, breaker open/close events, retry-budget exhaustion, and the adaptive limits — so you can see both kinds of trouble: a cascade leaking through, or throttling that is biting healthy traffic.

Operating rule: every caller retries sparingly within a strict retry budget; every dependency runs behind a bulkhead and a circuit breaker that trips fast and re-admits slowly via half-open probing; each node self-tunes its accepted load to a healthy latency/error band and sheds prioritized excess early; capacity scales on the slow loop. A slowdown then stays contained to the affected component, and the system walks itself back to full service in minutes — with manual traffic-shedding kept only as a never-needed last resort.

## Recommendation R3
The outages are driven by a self-reinforcing loop, not by raw demand: a few slow nodes cause timeouts, callers retry automatically, the retries multiply load on exactly the nodes least able to absorb it, and the overload spreads through shared dependencies until everything saturates. The design must therefore do four things at once — cut the loop's gain, isolate trouble before it spreads, bound how much work the system admits, and make recovery automatic — without throttling so hard that ordinary traffic suffers.

First, defang retries. Every client and internal caller should retry with exponential backoff plus random jitter, never on a fixed immediate timer, and each caller should hold a retry budget: retries may consume only a small fraction (for example 10 percent) of that caller's outbound requests over a rolling window. When the budget is exhausted, further failures fail fast rather than being retried. This single change removes most of the amplification, because it prevents a dip in success rate from being converted into a multiplied wave of new load.

Second, put a tripping interrupter on each caller-to-dependency path. Each caller tracks the recent error and latency rate to a given backend. When that rate crosses a danger threshold, the path "opens": calls fail fast (returning a cached value, a default, or a clear error) instead of piling onto a dying dependency and exhausting connection pools. Crucially, before fully opening, prefer to keep degraded-but-working service flowing — open only when continuing would clearly make things worse — so the interrupter does not turn a tolerable slowdown into a self-inflicted outage. Set the open threshold with hysteresis (a gap between the level that trips and the level that recovers) so the path does not flap rapidly between states.

Third, isolate by partition. Group backend nodes and the resources they share (connection pools, thread pools, queues) into bulkheads so that a hot spot consuming one partition's capacity cannot drain the capacity the rest of the system needs. A fault then stays confined to its compartment instead of propagating across the whole pool.

Fourth, bound admitted work against finite capacity. At the entry point and at each saturating service, enforce admission control: queues are bounded, and when a queue or concurrency limit is reached, the lowest-priority work is shed first (and shed early, before collapse), while high-priority and already-in-flight work is protected. Operate each service with deliberate headroom — a safety margin below its theoretical maximum — and treat that margin as the normal operating ceiling, not a number to be eliminated for efficiency. Bursts that fit inside the margin are absorbed; bursts that exceed it are shed at the edge, where shedding is cheap, rather than deep in the system where it cascades. Autoscaling can add capacity as a secondary cushion, but because it lags a fast retry storm it must not be the primary guard.

Fifth, make recovery self-testing. Recovery must not require an operator to manually shed traffic. After an interrupter opens, it waits a cooldown and then enters a trial state: it lets a single probe request (or a trickle) through. If the probe succeeds, it ramps traffic back gradually — not all at once — watching error and latency the whole way; if the probe fails, it re-opens and waits again. This staged, monitored re-entry is what lets a localized slowdown clear on its own within minutes: the loop gain is already gone, the fault was confined, and the system gently re-tests the path until it is genuinely healthy, ramping rather than slamming load back on so it does not immediately re-trip.

Tuning is where this succeeds or fails. The thresholds for tripping, shedding, and backoff must be calibrated so that ordinary variation — normal latency jitter, a routine traffic bump — does not trip them, while a genuine degradation does. Use the existing monitoring to drive this: trip on sustained error/latency excursions, not single spikes; require a signal to persist briefly before acting; and review false trips and missed trips regularly, retuning as load patterns change. Keep every control observable, so operators can see why traffic was slowed, capped, or interrupted, and can override in a true emergency.

The final operating rule: never let a failed request generate more total work than the original — degrade, shed the least important load early, and break a connection before retries can multiply it; confine every fault to its own compartment; run with deliberate headroom rather than at the edge; and recover by quietly probing and ramping a path back to health, never by slamming full load onto a node that has not yet proven it is well.

## Recommendation R4
The outages are not caused by the size of the demand spike; they are caused by how the system repays it. A few slow nodes time out, callers retry, the retries pile onto the already-slow nodes, more nodes tip over, and the disturbance regenerates itself until everything is saturated. The cure is to make the system unable to amplify a local slowdown, and to make its return automatic. Treat in-flight request count and queue depth as the vital signs you defend, and keep them inside fixed bounds at every node.

First, stop the retry storm, which is the engine of the cascade. Give every client and internal caller a strict, small retry budget — at most one retry, and only for requests that are safe to repeat — governed by a token bucket so that retries can never exceed a few percent of fresh traffic. Space retries with exponential backoff plus randomized jitter so that thousands of callers do not resynchronize into a second wave. A request that has already waited past its deadline must be abandoned, not retried: enforce deadline propagation so a caller never works on a response no one is waiting for.

Second, cut the coupling so one hot spot cannot recruit its neighbors. Give each backend node and each downstream dependency a bounded concurrency limit — a ceiling on simultaneous in-flight requests sized to leave headroom below the saturation point. When the ceiling is reached, the node sheds the excess immediately rather than queuing it indefinitely; queues must be short and bounded so latency cannot grow without limit. Isolate pools per dependency (separate connection and thread pools for each backend or database) so that a single slow dependency exhausts only its own pool and healthy paths keep flowing.

Third, fail fast on dependencies that are actually sick. Wrap each call to a backend or shared resource in a monitored cut-off: when its error rate or latency crosses a threshold over a short rolling window, the path opens and callers get an immediate error or a cached or degraded fallback instead of stacking up on a dying dependency. Scope each cut-off narrowly to one dependency, never globally, and add a gap between the trip threshold and the recovery threshold so it cannot rapidly flap between open and closed.

Fourth, when load still exceeds safe capacity, shed by priority rather than collapsing. At admission, classify traffic — health checks and high-value user requests first, retried and low-value background work first to be dropped. Under pressure the system rejects the least important load early and cheaply, protecting the throughput that matters. This is what keeps a hot spot localized: excess is refused at the edge instead of propagating inward.

Fifth, make recovery a staged, automatic process, not a manual flip. A cut-off path does not slam back to full traffic; after a cooldown it lets a single trial request through, and only on success does it ramp load back in stages — a small fraction, then more, watching error rate and latency at each step and stepping back down if the vital signs deteriorate. Concurrency limits and admission thresholds relax along the same gradient. Because shedding and cut-offs relieve the overloaded nodes the instant they trip, the nodes drain their backlogs within seconds, the trial probes succeed, and the ramp restores full service in minutes with no operator dumping traffic.

Tune against both failure modes at once. Set concurrency ceilings and trip thresholds from observed healthy latency with deliberate headroom, so they fire on genuine overload but tolerate ordinary variation — otherwise the safeguards themselves become the outage by throttling normal traffic. Keep the hysteresis gap and backoff jitter wide enough to prevent oscillation. Bounded autoscaling may add capacity for sustained higher load, but it is a slow secondary measure, never the primary defense, and never a substitute for the limits above; if peaks become permanent, raise the baseline deliberately.

Final operating rule: every caller retries at most a little and with jitter; every node and dependency runs under a hard concurrency limit with a short bounded queue and an isolated pool; every dependency call sits behind a narrowly scoped, hysteretic fast-fail with cached fallback; excess load is shed by priority at admission; and recovery is always a monitored probe-and-ramp, never an all-at-once reconnect. Hold the in-flight count inside its bounds and the cascade has nothing to feed on.

## Recommendation R5
The outage is not caused by the size of any spike; it is caused by the system feeding its own distress. When a few nodes slow, every failed call provokes automatic retries, and those retries land on the very nodes that are already struggling — so the response to overload becomes a fresh source of overload. Past a certain point the whole service flips into a saturated state that will not clear itself, because the load needed to keep it jammed is far lower than the load that first jammed it. The fix has two halves: stop the amplification before it tips, and give the system a way to climb back out on its own.

First, cut the amplification. Give every client and internal caller a retry budget — retries may consume only a small fraction of its successful-request volume (start near ten percent) — and have it stop retrying once that budget is spent. Use exponential backoff with randomized jitter so retries spread out instead of arriving in synchronized waves, and never retry requests that aren't safe to repeat or are already almost certain to fail. This alone keeps a slow node from being hammered by the traffic meant to recover from its slowness.

Second, put a guard in front of each backend that limits how much work it admits, and let that limit move with the node's own health. Each node continuously reports its latency, error rate, queue depth, and in-flight concurrency. An adaptive admission controller raises the allowed concurrency gently while the node is healthy and cuts it sharply the moment latency or errors climb — additive increase, multiplicative decrease — so admitted demand tracks what the node can actually serve. When demand still exceeds capacity, the guard sheds the excess by rejecting requests immediately with a clear "retry after" signal rather than letting them queue into slow timeouts that pin resources. A fast, cheap rejection protects far more than a slow success.

Third, contain blast radius. Give each downstream dependency its own isolated pool of connections and concurrency slots, so one hot dependency cannot starve the capacity that healthy paths need. Wrap each dependency in a breaker that opens when its error or latency crosses a limit, fails fast while open, and probes for health with a trickle of test traffic before reopening. A local problem is then walled off instead of draining the whole service.

The recovery half hinges on one deliberate asymmetry. Use two different thresholds: trip protection at one level, but only relax it after pressure has fallen well below a second, lower level. Because the jammed state is self-sustaining, simply returning load to the level that the node could "normally" handle will re-trip it instantly — which is exactly why manual recovery historically required shedding almost everything. By making the controller re-admit traffic only after the node has genuinely recovered headroom, and then ramp admission back up gradually through the breaker's half-open probe rather than all at once, the system walks itself out of the bad state in the same way an operator would, but in seconds and without a human. The gap between the two thresholds, plus a small dead-band, also stops the guard from flapping open and shut around the boundary.

Keep capacity elastic but bounded: hold a little headroom and autoscale quickly on sustained healthy demand, rather than trying to over-provision against the worst case. The point of the controls is that you no longer need to.

Finally, watch the controls themselves. Track goodput (useful work completed), the shed rate, retry ratios, and breaker state. Two failure directions matter equally: runaway retries or rising shed-with-low-goodput means the damping is too weak; routine rejection of healthy traffic means it is too strong. Tune the retry budget, the decrease factor, and the trip/re-arm gap against this signal, and keep an operator override only as a last-resort backstop, never as the normal recovery path.

Operating rule: admit only what the slowest healthy node can actually serve, never retry faster than you succeed, and only re-open the gates once pressure has dropped below the line — not merely back to it.

## Recommendation R6
The outages are a self-feeding loop: a few slow nodes cause timeouts, timeouts trigger automatic retries, retries multiply load on exactly the nodes that are already struggling, and the overload spreads until everything is saturated. Because the level of load that ignites the loop is far higher than the level at which it dies down on its own, the system cannot climb back out without a human throwing away nearly all traffic. The design therefore has two jobs that must be built together: starve the amplifying loop at the point where it regenerates, and give the system a staged, self-driven way back.

Start at the retries, because that is where a small fault becomes a flood. Replace blind immediate retries with capped retries that back off and add randomness: at most one or two retries per request, each after an exponentially growing, jittered delay so callers do not resynchronize into waves. Govern retries with a budget — a caller may spend retries on no more than a small fraction (for example 10%) of its recent requests; once that budget is exhausted it stops retrying and fails fast. A failed-fast call should return a cached or default answer where one exists, so that cutting the retry does not simply convert load into errors. This single change removes most of the multiplier.

Wrap every call to a backend or dependency in an automatic trip. Each caller continuously watches the error rate and latency of each dependency; when either crosses a threshold, the caller stops sending new requests to that dependency and immediately serves fallbacks instead of piling up on timeouts. Keep the trip scope narrow — per dependency and, where possible, per shard or partition — so one hot partition is isolated without removing healthy capacity. Crucially, the threshold to re-engage must be stricter than the threshold that tripped it, and re-engagement must be a single trial request (a probe) that is promoted only if it succeeds: this gap prevents the breaker from oscillating rapidly between open and closed, which is worse than either state.

Protect the servers from the inside too. Give each node a bounded concurrency limit and a bounded queue; when the queue is full, reject new work at the door rather than accepting it and slowing everything already in flight. Shed by priority: health checks, critical user paths, and internal control traffic get reserved capacity and are admitted first; bulk, low-priority, and best-effort traffic are the first to be rejected when the node is near its limit. Partition callers and workloads into separate pools (bulkheads) so a flood in one workload cannot consume the threads, connections, or memory that another depends on. The combination of bounded queues, concurrency caps, and priority admission ensures a node degrades gracefully — staying responsive to important work — instead of collapsing.

These guards must not strangle normal traffic, which is the opposite failure. Size every limit from observed normal peaks plus headroom, exempt health checks and critical paths from shedding, and make every trip, shed, and rejection visible in monitoring with its reason, so a too-tight limit is caught and retuned rather than silently degrading service.

Recovery should be automatic and gradual, never a manual cliff. When a node or dependency recovers, do not return it to full traffic at once. Ramp it: admit a small share of traffic, watch error rate and latency for a short interval, and only increase the share if the metrics hold; if they degrade, step back down automatically. Pair this with autoscaling that adds capacity when sustained healthy load approaches limits — within a ceiling, since hardware cannot grow without bound — so the staged ramp has somewhere to grow into. The ramp rate must stay below the measured spare headroom, or the act of returning traffic will reignite the very loop you just broke.

The final operating rule: never let a failed call generate more total load than the original call, and never return a recovering component to full load faster than it can be proven to absorb. Cut the retry multiplier at its source, isolate faults narrowly with hysteretic trips, shed and queue by priority so nodes degrade instead of dying, and bring everything back on a monitored ramp that backs off at the first sign of relapse. A local slowdown then stays a local slowdown, and the system climbs back on its own within minutes.
