# S2_service_reliability — Independent grading

Scenario: cascading failure in a high-traffic online service. A local slowdown is amplified by automatic retries into a self-feeding loop that saturates the whole service; recovery requires manual traffic-shedding. Design mechanisms that keep a local slowdown local AND let the system recover on its own — without throttling so hard that normal traffic is routinely degraded.

---

## Recommendation R1

1. Structural coherence: 10/10 — Organized around a single load-theoretic invariant ("add less load per unit of trouble than the trouble itself" / loop gain < 1), with every mechanism (retry cap, fast reject, bulkhead, breaker, ramp, deadband) derived from and tied back to that invariant.
2. Failure-mode anticipation: 10/10 — Names synchronized retry waves (jitter), retry fan-out (forbid nested retries), slow-death timeouts vs fast reject, breaker flapping, slamming a not-yet-well node, and the masking failure (safeguards hiding real capacity shortage).
3. Implementability: 9/10 — Mechanisms are all standard and concrete (retry budget, response headers, high-water marks, half-open probe, staged ramp) and buildable, though the "no nested retries" requires per-request retry-lineage tracking that is left unspecified.
4. Mechanism grounding: 10/10 — Dense with specific numbers: 10% retry budget, 50%-over-10s breaker trip, 80% trip / 60% resume, 10%→25%→50%→100% staged ramp, plus explicit deadband and rolling windows.
5. Stakeholder/equity fit: 9/10 — Explicit priority order (background, best-effort, then bulk shed first; essential traffic protected) so degradation falls on the least-critical traffic; robust, though it does not name reserved/exempt capacity for health checks as crisply as R2/R3.
6. Threshold/time-horizon awareness: 10/10 — Directly targets the ignite-vs-extinguish gap with separate shed and recover thresholds (trip 80 / resume 60), deadband against ordinary variation, and treats sustained shedding as a capacity alarm rather than normal operation — squarely the success criterion.
TOTAL: 58/60

## Recommendation R2

1. Structural coherence: 9/10 — Clean two-job framing ("starve the amplifying loop, give a staged self-driven way back") explicitly motivated by the ignite-level >> die-down-level gap, with mechanisms flowing logically from retries inward to nodes and back out via ramp.
2. Failure-mode anticipation: 9/10 — Covers resynchronized waves, fail-fast turning load into errors (mitigated by cached/default answers), breaker oscillation, and explicitly that ramp rate above spare headroom reignites the loop; slightly less on nested/fan-out retries than R1.
3. Implementability: 9/10 — Concrete and standard: bounded queue + concurrency cap, per-shard breaker scope, fallback serving, autoscaling with a ceiling; all readily buildable.
4. Mechanism grounding: 8/10 — Good specifics (1–2 retries, 10% budget, stricter re-engage threshold, single probe) but leans on relative/qualitative thresholds ("crosses a threshold", "size from normal peaks plus headroom") rather than concrete trip/resume numbers like R1.
5. Stakeholder/equity fit: 9/10 — Strong: reserved capacity for health checks, critical user paths, internal control traffic; bulk/best-effort shed first; exemptions made explicit and monitored so a too-tight limit is caught.
6. Threshold/time-horizon awareness: 9/10 — Opens by naming the ignite >> extinguish gap, uses hysteretic re-engage thresholds, and crucially insists "ramp rate must stay below measured spare headroom" — directly addressing the recovery tipping point.
TOTAL: 53/60

## Recommendation R3

1. Structural coherence: 9/10 — Crisp "feedback problem not capacity problem" thesis with a clear cut-amplifier / self-protect / stop-spread / recover / don't-over-correct structure, and names the missing balancing loop explicitly.
2. Failure-mode anticipation: 8/10 — Synchronized recovery surge (jitter and stagger ramps across nodes), unbounded-queue hiding collapse, breaker flapping (hysteresis); less coverage of nested-retry fan-out and the fail-fast-to-errors conversion than R1/R2.
3. Implementability: 9/10 — Highly concrete: admission controller on live latency/queue depth, token-bucket retry budget, bounded queues, per-dependency concurrency limits; standard and buildable.
4. Mechanism grounding: 9/10 — Strong specifics: 1 retry cap, ~10% token-bucket budget, 1%→10%→50%→100% staged ramp, latency-keyed (not fixed) limits so headroom auto-expands — a notably good grounding choice.
5. Stakeholder/equity fit: 8/10 — Priority shedding stated (health checks and high-value traffic dropped last, bulk first; protection never touches normal interactive traffic until necessary), but reserved-capacity allocation is less explicit than R2.
6. Threshold/time-horizon awareness: 9/10 — Latency-keyed limits expand headroom automatically, hysteresis (trip fast, recover slow), staged jittered ramps with hold periods, and explicit aim that under normal load breakers stay closed and admission rejects nothing — well-aligned to the non-degradation criterion.
TOTAL: 52/60

## Recommendation R4

1. Structural coherence: 8/10 — Coherent cut-the-loop / contain / shed-right / self-correct / degrade structure, but reads more as a well-ordered checklist than a single derived invariant; the organizing principle is implicit rather than foregrounded as in R1.
2. Failure-mode anticipation: 9/10 — Adds points others miss: idempotent writes with request keys for safe retries, degrade-not-fail to avoid provoking more retries, and monitoring both failure directions (cascade leaking through vs throttling biting healthy traffic).
3. Implementability: 8/10 — Mostly concrete (retry budget, bulkhead pools, bounded queue, half-open probe, idempotency keys) but the "adaptive concurrency limiter / healthy latency band" is described in principle without thresholds, leaving real tuning to the implementer.
4. Mechanism grounding: 7/10 — Has the 10% retry budget and 1–2 retry cap, but most other limits stay qualitative ("trip threshold", "healthy target band", "ramps gradually") with no concrete trip/resume numbers or ramp stages.
5. Stakeholder/equity fit: 8/10 — Classifies traffic by importance, sheds optional first, and argues sizing protects spare capacity so normal traffic is not routinely degraded; solid but reserved/exempt critical paths are less crisply named than R2.
6. Threshold/time-horizon awareness: 8/10 — Uses asymmetric hysteresis (trip fast, re-admit only after sustained health), half-open probing, and an adaptive limiter that finds the safe load level, plus fast-loop-vs-slow-autoscale time separation; strong on the recovery side but does not name the ignite-vs-extinguish gap as explicitly as R1/R2.
TOTAL: 48/60

---

## Comparison and ranking

All four correctly diagnose an amplification/feedback problem (not capacity), cut retries at the source with a ~10% budget plus jittered backoff, isolate with bulkheads and breakers, shed by priority, and recover via half-open ramps with hysteresis — substantively very close. They separate on grounding precision and on how directly they engage the ignite-vs-extinguish tipping point. R1 is strongest: it derives everything from one invariant (loop gain < 1), supplies the most concrete dual thresholds (trip 80 / resume 60, staged ramp), and uniquely treats sustained shedding as a capacity alarm. R2 most sharply frames the ignite >> die-down gap and the "ramp below spare headroom" recovery limit. R3's latency-keyed adaptive limits are an elegant non-degradation choice. R4 adds idempotency and degrade-not-fail but stays most qualitative on thresholds.

Best-to-worst: R1 > R2 > R3 > R4.

Would adopt: R1 — most coherent, most concretely grounded, and most directly engineered against both failure modes (cascade and over-throttling).
