# Grader 1 — S2_service_reliability

Independent scoring. Six dimensions, 1-10. Substance only.

---

## Recommendation R1
1. Structural coherence: 9/10 — The three-point attack (caller throttling, per-dependency isolation, staged self-driven recovery) is tightly interlocked: remove the bulkhead and breakers leak across pools, remove the retry budget and the staged ramp re-ignites instantly, so the pieces are genuinely load-bearing on one another.
2. Failure-mode anticipation: 9/10 — Names retry storms, thread/connection exhaustion, unbounded queues hiding backlog, synchronized retry waves, flapping near the threshold, and premature reconnect re-lighting the fire, each with a specific counter-measure.
3. Implementability: 8/10 — Every mechanism (retry caps, budgets, breakers, bulkheads, priority classes, staged ramp) is standard and buildable with the stated levers, though wiring idempotency keys and per-class fairness across all callers is real work.
4. Mechanism grounding: 9/10 — Concrete throughout: 1-2 retries, 5-10% budget, retry-after signal, bounded fixed slots, hysteresis band with separate start/stop thresholds, single trial request then staged percentage ramp.
5. Stakeholder/equity fit: 9/10 — Explicit traffic classing (essential reads/writes protected, speculative/batch/internal sacrificed first), plus fairness within a class so no tenant or region is starved, and a rule that inability to serve the protected class signals capacity not deeper shedding.
6. Threshold/time-horizon awareness: 9/10 — Acts at an early-warning threshold "before saturation, not at the cliff," with hysteresis to avoid flap and an explicit fast-exit/slow-gated-return asymmetry tied to the fragile re-engagement moment.
TOTAL: 53/60

## Recommendation R2
1. Structural coherence: 9/10 — Retry budget, bulkhead, breaker, adaptive concurrency limiter, and autoscaling form a coherent fast-loop/slow-loop stack where each layer depends on the one beneath; pulling the budget would let admission control rebound into a storm.
2. Failure-mode anticipation: 8/10 — Covers retry storms, thread/connection/memory starvation, queuing behind dying nodes, flapping, and "throttling biting healthy traffic," each countered; slightly less vivid on duplicate-work and synchronized-wave detail than R1.
3. Implementability: 7/10 — Mostly standard, but the per-node adaptive concurrency limiter tracking a healthy latency/error band is the hardest piece to tune correctly and is asserted more than specified.
4. Mechanism grounding: 8/10 — Good concreteness (10% budget, half-open probe, hysteresis, bounded queue with admission control, idempotency keys) but the adaptive limiter's actual control law is left as "raise when healthy, lower when not."
5. Stakeholder/equity fit: 7/10 — Classifies traffic by importance and protects critical over optional, and sizes shedding to protect spare capacity, but does not address fairness within a class or which tenants/regions absorb the shed.
6. Threshold/time-horizon awareness: 8/10 — Asymmetric thresholds, trip-fast/re-admit-slow, half-open probe and gradual ramp, autoscaling as the slow loop; engages the tension well but does not name the bistable tipping point as sharply as R5/R6.
TOTAL: 47/60

## Recommendation R3
1. Structural coherence: 8/10 — Four explicit jobs (cut loop gain, isolate, bound admitted work, automatic recovery) with headroom as the unifying operating ceiling; coherent, though the "prefer degraded-but-working over opening the breaker" instruction sits a little in tension with fail-fast and is not fully reconciled.
2. Failure-mode anticipation: 8/10 — Connection-pool exhaustion, flapping, false trips on noise, autoscaling lag behind a retry storm, and the self-inflicted-outage risk of over-eager breakers are all named with counters.
3. Implementability: 8/10 — Standard mechanisms plus an explicit tuning-and-review loop (false-trip/missed-trip review) that makes operation realistic; nothing exotic.
4. Mechanism grounding: 8/10 — 10% budget, hysteresis gap, bounded queues, edge shedding, single-probe-then-ramp, persistence requirement before acting; strong, though "open only when continuing would clearly make things worse" is judgment rather than a rule.
5. Stakeholder/equity fit: 7/10 — Priority shedding with high-priority and in-flight work protected and excess shed at the cheap edge, but no within-class fairness or tenant/region allocation.
6. Threshold/time-horizon awareness: 9/10 — Strong emphasis on deliberate headroom as the normal ceiling (not to be optimized away), edge-vs-deep shedding, hysteresis, persistence-before-acting, and autoscaling-lags-the-storm; squarely on the non-linear tension.
TOTAL: 48/60

## Recommendation R4
1. Structural coherence: 9/10 — Organized around defending in-flight count and queue depth as "vital signs," with retry budget, per-dependency concurrency limits, isolated pools, scoped fast-fail, priority shedding, and probe-and-ramp all converging on that single invariant; removing any one breaks the bound.
2. Failure-mode anticipation: 9/10 — Retry storms, synchronized second waves, working on responses no one awaits (deadline propagation), pool exhaustion, unbounded queue latency growth, flapping, and "safeguards themselves becoming the outage" are each named and countered.
3. Implementability: 8/10 — All mechanisms are concrete and standard; deadline propagation and per-dependency isolated pools are real effort but clearly specified and within the stated levers.
4. Mechanism grounding: 9/10 — Very concrete: token-bucket retry at a few percent, at most one retry, deadline abandonment, concurrency ceilings sized from observed healthy latency with headroom, short bounded queues, narrowly scoped hysteretic fast-fail, staged probe-and-ramp.
5. Stakeholder/equity fit: 7/10 — Clear priority ordering (health checks and high-value first, retried/background dropped first) shed cheaply at the edge, but no within-class fairness or tenant/region distribution of the cost.
6. Threshold/time-horizon awareness: 8/10 — Headroom-sized ceilings, hysteresis gap, jitter against oscillation, autoscaling as slow secondary, and "if peaks become permanent raise the baseline"; engages both directions well but states the bistable gap less explicitly than R5/R6.
TOTAL: 50/60

## Recommendation R5
1. Structural coherence: 8/10 — Two-halves framing (cut amplification, climb back out) with retry budget, AIMD adaptive admission, isolated pools, breaker, and the two-threshold asymmetry all serving the bistable-state model; coherent and focused, though thinner on caller-side mechanics than peers.
2. Failure-mode anticipation: 7/10 — Names the self-sustaining jam, slow timeouts pinning resources, flapping around the boundary, and the symmetric over/under-damping failure; fewer distinct modes enumerated than R1/R4 (e.g., no synchronized-wave or deadline detail beyond jitter).
3. Implementability: 7/10 — AIMD admission control and the two-threshold re-arm are buildable, but tuning the gap and decrease factor against goodput is non-trivial and the design leans heavily on getting that adaptive controller right.
4. Mechanism grounding: 8/10 — AIMD (additive-increase/multiplicative-decrease) is the most specific control law in the set, plus 10% budget, dead-band, half-open probe, and goodput/shed/retry-ratio monitoring; some limits left as "a little headroom."
5. Stakeholder/equity fit: 6/10 — Sheds excess with a clear retry-after and protects goodput, but barely engages which traffic classes or tenants bear the cost — least explicit on allocation in the set.
6. Threshold/time-horizon awareness: 10/10 — Best in set: explicitly models the bistable tipping point (jamming load far above clearing load), explains why returning to the "normal" level re-trips instantly, and prescribes re-opening only after pressure drops below a lower line plus dead-band — directly the hysteresis/cascade tension.
TOTAL: 46/60

## Recommendation R6
1. Structural coherence: 9/10 — Two jointly-built jobs (starve the loop, staged self-driven return) with capped+budgeted retries, narrow hysteretic trips, bounded concurrency/queue with priority admission, bulkheads, and a ramp rate held below spare headroom; the ramp-below-headroom rule ties recovery directly to the containment mechanisms.
2. Failure-mode anticipation: 9/10 — Retry storms, synchronized waves, converting load into errors (counters with cached fallback), breaker oscillation ("worse than either state"), thread/connection/memory contention, and re-igniting the loop by ramping too fast are each named and countered.
3. Implementability: 8/10 — All standard mechanisms with explicit per-shard/partition trip scope and a monitoring-with-reason requirement; realistic to execute within the levers, with normal effort.
4. Mechanism grounding: 9/10 — 1-2 retries, 10% budget, per-dependency/per-shard trip scope, stricter re-engage threshold, single trial probe, reserved capacity for critical/health/control traffic, ramp rate below measured spare headroom, ceiling on autoscaling.
5. Stakeholder/equity fit: 8/10 — Strongest priority scheme: reserved capacity for health checks, critical user paths, and internal control traffic, with bulk/best-effort shed first and exemptions made explicit; lacks only within-class fairness/tenant allocation.
6. Threshold/time-horizon awareness: 9/10 — Explicitly states the ignition load far exceeds the die-down load, requires re-engage threshold stricter than trip, and pins the ramp rate below measured spare headroom so return cannot reignite the loop; very close to R5 minus the dead-band formalism.
TOTAL: 52/60

---

## Comparison (~150 words)

All six share the same correct diagnosis (the cascade is self-feeding, not demand-driven) and the same core toolkit: bounded retries with budgets and jitter, per-dependency breakers, bulkheads, priority shedding, and staged probe-and-ramp recovery. They separate on depth, not direction. R1 is the most complete and well-rounded: richest failure catalog, explicit within-class fairness (unique in the set), early-warning headroom, and a clean fast-exit/slow-return asymmetry. R6 nearly matches it and adds the sharpest recovery invariant — ramp rate below measured spare headroom — plus the best priority/reserved-capacity scheme. R4 is tight and concrete around its "in-flight count" invariant and uniquely includes deadline propagation. R5 owns the theory of the bistable tipping point most deeply but is thinnest on equity/allocation and breadth. R3 and R2 are solid but somewhat more generic; R2's adaptive limiter is asserted more than specified. None fully addresses tenant/region cost distribution except R1.

## Ranking (best to worst)
R1 > R6 > R4 > R3 > R2 > R5

## Would adopt
R1 — most structurally complete, best failure-mode catalog, and the only one that engages within-class fairness alongside a correctly asymmetric, self-driven recovery; R6 is the close runner-up and its "ramp below measured spare headroom" rule is worth folding into R1.
