# S2_service_reliability — Grader 2

Scored on substance only. Six dimensions, 1-10, one-sentence justification each.

---

## Recommendation R1
1. Structural coherence: 9/10 — The four layers (caller throttle, per-dependency breaker+bulkhead, priority shedding, staged recovery) are explicitly load-bearing and interlocking; remove the bulkhead and a slow node again drains the whole thread budget, remove staged recovery and the breaker just re-lights the fire.
2. Failure-mode anticipation: 9/10 — Names retry resynchronization waves, unbounded hidden queues, premature reconnect re-igniting the fire, flapping near the line, and "fresh capacity behind ungoverned retries consumed instantly," each with a named counter (jitter, bounded deadline-aware queues, single trial request, hysteresis, demand-governing-before-scaling).
3. Implementability: 9/10 — Every mechanism (retry budget, idempotency keys, bulkhead slot counts, retry-after, hysteresis band, fault-injection drills) is standard and fits the stated levers; the manual-drain-as-defect framing is a concrete operational rule.
4. Mechanism grounding: 9/10 — Concrete thresholds and rules throughout (1-2 retries, 5-10% budget, single trial request then staged percentages, separate start/stop thresholds, reject-don't-park) rather than aspiration.
5. Stakeholder/equity fit: 9/10 — Explicit traffic classes (essential reads/writes protected, speculative/batch/internal sacrificed first), fairness within a class so no tenant/region starves, and the rule that inability to serve the protected class is the signal to add capacity not shed deeper.
6. Threshold/time-horizon awareness: 8/10 — Early-warning "act before saturation, not at the cliff," headroom margin, hysteresis band, and fast-exit/slow-gated-return all engage the tipping point, though it doesn't quite articulate the self-sustaining-jam asymmetry (low hold-load vs high trip-load) as sharply as R5/R6.
TOTAL: 53/60

## Recommendation R2
1. Structural coherence: 9/10 — Retry budget, bulkhead, breaker, prioritized admission, adaptive concurrency limiter, and autoscaling form a clean fast/slow-loop hierarchy where each piece's failure is named (no budget = retry storm; no bulkhead = thread starvation).
2. Failure-mode anticipation: 8/10 — Covers resynchronized waves, flapping, retries rebounding off shed signals, and explicitly monitors both directions (cascade leaking through vs throttling biting healthy traffic), though slightly less vivid on the re-engagement fragility than R1.
3. Implementability: 8/10 — Adds an adaptive concurrency limiter on top of the standard kit, which is realistic but more tuning-heavy; all within stated levers.
4. Mechanism grounding: 8/10 — Strong concrete rules (10% budget, half-open probe trickle then ramp, healthy-target latency band, asymmetric thresholds) with the adaptive limiter being directionally specified rather than numerically.
5. Stakeholder/equity fit: 7/10 — Classifies traffic by importance and protects critical over optional, but doesn't address fairness within a class (no tenant/region starvation) the way R1 does.
6. Threshold/time-horizon awareness: 8/10 — Asymmetric trip-fast/re-admit-slow, half-open probing, and fast-loops-hold-before-capacity-arrives engage the tipping point well, though the cascade asymmetry is implied rather than named.
TOTAL: 48/60

## Recommendation R3
1. Structural coherence: 8/10 — Four-then-five-part structure (cut loop gain, interrupter, bulkhead, admission control, self-testing recovery) coheres, and the "degrade rather than self-inflict outage" nuance ties breaker behavior to shedding logic.
2. Failure-mode anticipation: 8/10 — Names interrupter turning a tolerable slowdown into self-inflicted outage, flapping, single-spike false trips vs missed trips, and autoscaling lagging a retry storm, each countered (open only when continuing makes it worse, hysteresis, persistence requirement, headroom as primary guard).
3. Implementability: 8/10 — Adds an explicit ongoing tuning/review discipline (review false and missed trips, retune as load shifts) which is realistic and executable within the team's monitoring.
4. Mechanism grounding: 8/10 — Concrete (10% budget, bounded queues, shed lowest-priority early at the edge, hysteresis gap, single probe then ramp, require signal to persist briefly) though some thresholds left to be calibrated.
5. Stakeholder/equity fit: 7/10 — Protects high-priority and already-in-flight work, sheds lowest-priority first at the cheap edge, but no within-class fairness guarantee.
6. Threshold/time-horizon awareness: 8/10 — "Shed at the edge where cheap rather than deep where it cascades," deliberate-headroom-as-normal-ceiling, autoscaling-lags-so-not-primary, and persistence-before-acting all engage the non-linear boundary thoughtfully.
TOTAL: 47/60

## Recommendation R4
1. Structural coherence: 9/10 — Organized around defending in-flight count and queue depth as "vital signs," with retry budget, per-dependency concurrency limits + isolated pools, narrowly-scoped fast-fail, priority admission, and probe-and-ramp recovery all hanging off that invariant.
2. Failure-mode anticipation: 9/10 — Names retry storm, neighbor recruitment, unbounded queue latency growth, pool exhaustion, flapping, working on responses no one awaits, and safeguards-becoming-the-outage, each countered (token bucket, bounded short queues, isolated pools, hysteresis gap, deadline propagation/abandonment, headroom-calibrated thresholds).
3. Implementability: 9/10 — All mechanisms are concrete and standard (token-bucket retry budget, deadline propagation, per-dependency pools, narrow cut-offs); deadline propagation is the one item needing cross-service plumbing but is well within "change client and server behavior."
4. Mechanism grounding: 9/10 — Among the most concrete: at-most-one-retry, token bucket, abandon-past-deadline, hard concurrency ceilings sized below saturation, short bounded queues, narrowly scoped hysteretic fast-fail, probe-then-staged-ramp with step-down.
5. Stakeholder/equity fit: 7/10 — Clear priority classes (health checks and high-value first, retried/background dropped first) and protecting in-flight work, but no within-class fairness across tenants/regions.
6. Threshold/time-horizon awareness: 8/10 — Concurrency ceilings sized below saturation with headroom, hysteresis gap, autoscaling-lags-so-secondary, and probe-and-ramp engage the tipping point; the self-sustaining-jam asymmetry is implied via "relieve the instant they trip" rather than stated.
TOTAL: 51/60

## Recommendation R5
1. Structural coherence: 8/10 — Tight two-half framing (stop amplification / climb back) with retry budget, adaptive AIMD admission controller, per-dependency bulkhead+breaker, and a two-threshold recovery asymmetry that is the structural spine; somewhat thinner on the queue/bulkhead internals than R4/R6.
2. Failure-mode anticipation: 8/10 — Names slow-timeout queuing pinning resources, re-trip on returning load to the "normal" level, flapping around the boundary, and explicitly the two tuning directions (too-weak vs too-strong damping), each countered (fast cheap rejection, two-threshold gap + dead-band, goodput-driven tuning).
3. Implementability: 7/10 — AIMD adaptive admission control is the most sophisticated single mechanism here and demands careful tuning; achievable but the heaviest lift, and node internals (queues, bulkheads) are less spelled out.
4. Mechanism grounding: 8/10 — Concrete on additive-increase/multiplicative-decrease, retry-after rejection, two-threshold trip/re-arm gap, dead-band, and goodput/shed-rate/retry-ratio signals, though admission limits are adaptive rather than fixed numbers.
5. Stakeholder/equity fit: 6/10 — Sheds excess and protects goodput, but does the least on traffic-class allocation — no explicit priority tiers and no within-class fairness, so "who gets degraded" is least developed.
6. Threshold/time-horizon awareness: 10/10 — Best in the set on the tipping point: explicitly states the jammed state is self-sustaining because hold-load is far below ignite-load, derives the two-threshold ("below the line, not merely back to it") recovery directly from that asymmetry, and frames the whole recovery as walking out of the bad state the way an operator would.
TOTAL: 47/60

## Recommendation R6
1. Structural coherence: 9/10 — Two-jobs framing (starve the loop / staged return) with retry budget+fallback, narrow hysteretic per-shard breaker, bounded concurrency+queue with priority admission, bulkheads, and ramp-below-headroom recovery; pieces are explicitly mutually dependent (ramp rate must stay below spare headroom or it reignites the loop).
2. Failure-mode anticipation: 9/10 — Names resynchronized waves, cutting-retry-merely-converting-load-to-errors, breaker oscillation ("worse than either state"), node collapse from accepting work, and too-tight limits silently degrading service, each countered (jitter, cached fallback on fail-fast, stricter re-engage threshold + single probe, reject-at-door, visible-reasoned monitoring + retune).
3. Implementability: 9/10 — All standard, well-scoped mechanisms; per-shard/partition breaker scope and reserved capacity for critical paths are concrete and executable within the stated levers.
4. Mechanism grounding: 9/10 — Concrete throughout (1-2 retries, 10% budget, per-shard trip scope, bounded queue reject-at-door, reserved capacity for critical traffic, ramp-rate-below-spare-headroom, autoscale within a ceiling).
5. Stakeholder/equity fit: 8/10 — Strongest priority allocation after R1: reserved capacity for health checks/critical user paths/control traffic, bulk and best-effort shed first, and exempting critical paths from shedding; still no explicit within-class tenant/region fairness.
6. Threshold/time-horizon awareness: 9/10 — Opens by naming the ignite-load >> die-down-load asymmetry, ties recovery to "ramp rate must stay below measured spare headroom or you reignite the loop," and uses stricter re-engage threshold + probe; nearly as sharp as R5 on the tipping point while also covering node internals.
TOTAL: 53/60

---

## Comparison and ranking

All six diagnose the same root cause (retry amplification, not raw demand) and converge on the same canonical toolkit: retry budgets with jittered backoff, per-dependency bulkheads, circuit breakers, priority load-shedding, hysteresis, and probe-and-ramp recovery. They differ at the margins. R1 and R6 are the most complete: both lock the pieces into explicit interdependencies, develop priority/equity allocation furthest (R1 alone adds within-class fairness so no tenant or region starves; R6 adds reserved capacity for critical traffic), and treat recovery fragility seriously. R5 is the sharpest single insight — it alone rigorously derives the two-threshold recovery from the self-sustaining-jam asymmetry (hold-load far below ignite-load) — but pays for that focus with the weakest equity treatment and the heaviest, least-spelled-out mechanism (AIMD). R4 is the most operationally concrete and disciplined (deadline propagation, token buckets, vital-signs framing) but slightly behind R1/R6 on equity. R2 and R3 are solid and complete but a notch less vivid on re-engagement fragility and within-class fairness.

### Best to worst
1. R1 (53)
2. R6 (53)
3. R4 (51)
4. R2 (48)
5. R3 (47)
6. R5 (47)

R1 and R6 tie at 53; R1 edges ahead on equity (within-class fairness) and the cleanest "manual drain = defect" operating discipline.

### Would adopt: R1
It is the most structurally complete and the only one that guarantees fairness within a protected class while keeping every mechanism concrete, executable within the stated levers, and validated by fault-injection drills. I would graft R5's explicit "re-open below the line, not back to it" two-threshold framing onto R1's recovery stage to close its one relative gap.
