# Working — Congestion-collapse / retry-storm design (control policy run)

## STATE.problem
- statement: a modest load spike slows a few backend nodes; clients and internal callers auto-retry failed requests; retries multiply load on already-struggling nodes; more nodes fall over; the whole service saturates; recovery needs manual traffic-shed + slow restart.
- constraints: good monitoring; can change client+server behavior, set limits, add capacity; cannot indefinitely over-provision.
- success: a local slowdown stays local (no full saturation) AND the system self-recovers without manual shedding within minutes.
- failure: a single hot spot can still cascade to total outage, OR safeguards throttle so aggressively that normal traffic is routinely degraded.

## Verbs run (verb — why)

1. **match (retrieve)** — `search_prime` with the domain-stripped meta-model (self-reinforcing positive-feedback amplification from local overload to global saturation, recovery needs external load removal) + `search_by_facets` over six structural facets, because the problem is oblique: it instantiates an amplifying loop, an isolation need, a self-recovery need, a demand-shedding need, and a saturation-threshold nonlinearity at once. Facet retrieval surfaced what a blended query buries.
   - Prime hits: `cascade`/`critical_mass`/`threshold_and_criticality` (the failure structure), `circular_causality` + `system_archetypes` (the reinforcing loop), `fail_safe`, `systemic_risk` (isolation facet), `homeostasis`/`controlled_reentry` (self-recovery facet), `load_balancing`/`buffering`/`system_slack` (shedding facet).
   - Archetype hits: `saturation_avoidance`, **`load_shedding`**, **`fault_tolerant_operation`**, `equilibrium_restoration`, `balancing_loop_stabilization`, `instability_dampening`.
2. **salience-rank** — ranked candidates by whether removing them changes the recommendation.
   - load_bearing: `circuit_breaker` (structural; the trip-isolate-hold-reset unit that both breaks the retry loop AND is the recovery gate), `fault_tolerant_operation` (the runtime posture that keeps local fault from going global), `load_shedding` (priority admission/shedding so capacity is protected), `controlled_reentry` (staged autonomous recovery, asymmetric to exit).
   - context: `cascade`/`systemic_risk`/`circular_causality` (diagnose the disease, not the cure), `equilibrium_restoration`/`balancing_loop_stabilization` (frame the bounded-counterforce + no-overshoot constraint), `saturation_avoidance` (early-threshold framing folded into the headroom rule).
3. **prune** — kept the 4 load-bearing primes + 2 framing archetypes. Cut `homeostasis` (subsumed by the balancing-loop framing), `buffering`/`system_slack` (queueing is part of the disease here — unbounded queues hide the backlog; I keep queues *bounded* instead), `saturation_avoidance` as a standalone (its move is the early warning threshold, already inside the design).
4. **compose** — typed relational model below.
5. **lift L1 (situation → skeleton)** — meta-model below; witness-checked, no domain terms.
6. **lift L2 (coverage-enforcement)** — coverage map below against the operative pattern.
7. **evaluate-fit (gate)** — anti-signature walk below.
8. **reconcile** — concrete model vs meta-model give one divergent inference (resolved below), confirming the meta-model is not decorative.

## STATE.model (compose — referential integrity)
Nodes: `caller`(actor), `retry_policy`(mechanism), `breaker`(mechanism), `node`(component), `admission_ctrl`(mechanism), `priority_class`(label), `health_signal`(signal), `recovery_ctrl`(mechanism), `autoscaler`(mechanism), `protected_fn`(function).
Edges (every edge references declared nodes; every operative prime annotated):
- caller →[issues, prime:circular_causality] node
- caller →[retries_on_fail, prime:cascade/circular_causality] retry_policy
- retry_policy →[amplifies_load_onto] node
- breaker →[wraps_calls_to, prime:circuit_breaker] node
- breaker →[trips_on, prime:circuit_breaker] health_signal
- admission_ctrl →[sheds_by, prime:load_shedding] priority_class
- admission_ctrl →[protects, prime:fault_tolerant_operation] protected_fn
- health_signal →[observes] node
- recovery_ctrl →[half_open_probes, prime:controlled_reentry] breaker
- recovery_ctrl →[ramps, prime:controlled_reentry] node
- autoscaler →[adds_capacity_for] node
- node →[isolation_boundary, prime:fault_tolerant_operation] protected_fn
Referential integrity: OK — all 4 load-bearing primes (circuit_breaker, load_shedding, fault_tolerant_operation, controlled_reentry) plus the two framing primes (circular_causality/cascade) appear as annotations; no dangling endpoints.

## STATE.meta_model (lift L1) — witness in model
- roles: amplifier-under-stress {caller+retry_policy}; protected-but-finite channel {node/protected_fn}; sensor {health_signal}; fast protective interrupter {breaker}; priority gate {admission_ctrl}; staged re-engager {recovery_ctrl}; bounded counter-capacity {autoscaler}.
- relations: reaction-to-failure increases failure-pressure (reinforcing loop); interrupter decouples on threshold; gate sacrifices low-value flow to preserve high-value flow; re-engager is asymmetric to the exit and probes before committing.
- invariants: local fault must not become global; corrective force bounded (no chronic over-throttle); recovery autonomous; queues bounded (no hidden backlog).
- No domain-specific terms remain (no "HTTP", "server", "API"). Witness-check passes: each role/relation/invariant maps to a model node/edge above.

## COVERAGE MAP
Operative pattern: **Fault-Tolerant Operation** (the runtime posture the design relies on), with **Circuit Breaker** (loop-break + recovery gate), **Load Shedding** (admission), and **Controlled Reentry** (recovery) supplying load-bearing sub-mechanisms. I map FTO's components; where a sub-pattern owns a component I name it.

| Component (FTO) | Status | Witness / reason |
|---|---|---|
| Critical Function Map | **instantiated** | Tag traffic into priority classes; user-facing reads/writes = protected fn; speculative/batch/low-priority = sheddable first. |
| Fault Model | **instantiated** | Named tolerated faults: a few nodes slowed by a load spike + retry amplification. Out of scope: total datacenter loss. |
| Fault Detection Signal | **instantiated** | Existing monitoring → per-dependency error-rate + latency (p99) + concurrency; the breaker's trip signal and the health_signal node. |
| Fault Isolation Boundary | **instantiated** | Per-dependency circuit breaker + bulkhead (bounded concurrency pool per downstream) so a slow node can't exhaust caller threads/connections. |
| Continuation Mode | **instantiated** | Degraded mode: fail-fast with cached/default fallback when breaker is open; shed low-priority classes; serve essential reads. |
| Compensation or Bypass Path | **instantiated** | Fallback responses + load-balancer eject of unhealthy node + bounded capacity add (autoscaler) — bypass around the failed element. |
| State Consistency Guard | **homolog** | Writes use bounded retry with idempotency keys; no silent retries on non-idempotent ops; shed work is explicitly rejected (clear error), not silently dropped — discharges "don't corrupt/duplicate state." |
| Recovery Policy | **instantiated** | Controlled Reentry: half-open single-probe → staged traffic ramp (e.g. 1%→10%→50%→100%) with health gates; re-trip on regression. Autonomous, asymmetric to the trip. |
| Redundancy Pool (opt) | **correctly_declined** | Problem bars indefinite over-provisioning; design protects existing capacity rather than buying standby pools. Bounded autoscale covers genuine demand; forcing a redundancy pool would distort the "can't over-provision" constraint. |
| Quorum or Voting Rule (opt) | **correctly_declined** | No replicated-decision/agreement subproblem; this is overload, not Byzantine/disagreement. Forcing it adds machinery the problem doesn't need. |
| Operator Override Protocol (opt) | **homolog** | Manual kill-switch / "drain" remains available as escalation, but success criteria demand autonomy — override is the exception path, not the normal loop. Discharges "human authority under fault" without making manual shedding the design. |
| Observability Instrumentation (opt) | **instantiated** | Reuses existing good monitoring; adds breaker-state + shed-rate + ramp-stage dashboards. |
| Fault-Tolerance Test Case (opt) | **instantiated** | Load/fault-injection drill (force a hot node, confirm containment + autonomous recovery, confirm normal traffic not throttled). |

Cross-pattern load-bearing components also covered:
- **Circuit Breaker** {trip threshold / isolation scope / reset policy + hysteresis half-open}: instantiated (error+latency threshold; per-dependency scope; timed half-open reset with hysteresis to prevent chattering).
- **Load Shedding** {priority rule, explicit handling of shed work, bounded queue, recovery possible}: instantiated (priority classes; reject-with-clear-error; bounded queues with deadline-based drop; auto re-admit on recovery — addresses oscillatory-shedding + hidden-backlog failure modes).
- **Controlled Reentry** {staged progression, telemetry gates, graduated load, reversibility, deferred normality}: instantiated (the ramp); **the retry-storm root cause is also instantiated** via client-side change: bounded retries + exponential backoff with jitter + retry budget (cap retries to a small fraction of requests) + server "retry-after" backpressure — this is the move that actually breaks the reinforcing loop.

**No component is `missing`.**

## evaluate-fit (anti-signature gate)
- FTO "predictable variation, not real failure → use robustness_margin" — NOT tripped: nodes genuinely fail/saturate. keep.
- FTO "main move is backup capacity before failure → redundant_backup" — NOT tripped: design is runtime continuation, not standby provisioning. keep.
- FTO "safest response is to stop / fail-safe default" — NOT tripped: stopping the whole service is the failure we're preventing; we want bounded continuation. keep.
- FTO "can't detect or bound the fault" — NOT tripped: monitoring is good; faults are bounded (overload). keep.
- Circuit Breaker T5 "breaker converts degradation into hard outage" — addressed: breaker pairs with fallback + per-dependency scope so a slow node degrades to fallback, not global open. keep.
- Load Shedding "shedding too little/too late, or unfair/starvation/oscillatory" — addressed: early warning threshold (headroom, not max), priority+fairness across classes, hysteresis bands to stop flapping. keep.
- Equilibrium Restoration "overcorrection risk / target balance unjust" — addressed: bounded counterforce + ramp + hysteresis prevents the over-throttle failure mode; no unjust status quo here. keep.

## reconcile (concrete vs meta-model)
Concrete view says "open the breaker and fail fast." Meta-model view says "the reinforcing loop is the disease; an interrupter alone, with naive client retries, just moves the storm to the fallback/queue." Divergence resolved toward the reading consistent with both: the breaker is necessary but **insufficient** — it must be paired with client-side retry governance (backoff+jitter+budget) and server-side backpressure, or the amplifier survives the interrupter. This is the non-decorative payoff of the meta-model.

COST: ~9 operations (1 corpus_stats, 1 search_prime, 1 search_by_facets, 4 get_archetype, 2 get_prime), ~10000 tokens.

===RECOMMENDATION_BEGIN===
The outages are not really caused by the demand spike; they are caused by the system's own reaction to the spike. When a few nodes slow down, callers retry, and those retries pile fresh load onto the exact nodes that are already struggling, which slows them further and recruits their neighbors. The fix has to attack that self-feeding loop at three points: stop callers from amplifying trouble, stop a sick node from poisoning the healthy ones, and let the system climb back on its own.

Start at the caller, because that is where the amplification lives. Replace unconditional retries with disciplined ones: at most one or two retries, each after an exponentially growing delay with random jitter so callers do not resynchronize into waves. Cap retries with a budget — a caller may spend retries on no more than roughly five to ten percent of its requests in any window; once the budget is exhausted it fails fast instead of hammering. Make retries respect server backpressure: when a node is overloaded it returns an explicit "busy, do not retry yet" signal with a retry-after hint, and callers obey it. Only idempotent operations may be retried, and those carry idempotency keys so a retry never duplicates work.

Around every call to a backend dependency, wrap a circuit breaker. It watches that dependency's error rate and tail latency; when either crosses a threshold the breaker opens and calls fail immediately, returning a cached or default response rather than queuing behind a dying node. Crucially, scope each breaker to a single dependency and give each dependency its own bounded concurrency pool (a bulkhead) with a small fixed number of in-flight slots and bounded, deadline-aware queues. A slow node can then fill only its own pool; it cannot consume the caller's entire thread or connection budget and drag everything down. Requests that cannot get a slot are rejected quickly with a clear error — never silently dropped, never parked in an unbounded queue that hides a growing backlog.

When overall pressure still rises, shed load by priority rather than at random. Tag traffic into classes: essential user-facing reads and writes are protected; speculative, batch, and low-value internal calls are sacrificed first. Trigger shedding from an early-warning threshold that preserves headroom — act before saturation, not at the cliff — and use separate, wider thresholds for starting and stopping shedding (a hysteresis band) so the system does not flap on and off near the line. Shedding must be fair within a class so no single tenant or region is starved, and it must always leave the protected class genuinely served; if it cannot, that is the signal to add capacity, not to shed deeper.

Recovery must be a designed, staged process, not a hope. Exit from trouble is fast and blunt; return is slow and gated, because the moment of re-engagement is when the system is most fragile and a premature reconnect simply re-lights the fire. After a breaker has been open for a cooldown, let exactly one trial request through. If it succeeds, restore traffic in stages — a small percentage first, then more — advancing only while error rate and latency stay healthy, and dropping back a stage the instant they regress. Tie this to bounded, automatic capacity scaling: add nodes in response to sustained healthy load to meet real demand, but never as a substitute for governing demand, since fresh capacity behind ungoverned retries is consumed instantly.

Tune so that normal traffic is never routinely degraded: thresholds sit above ordinary peak behavior with margin, breakers trip on genuine distress rather than noise, and shedding touches only sacrificeable classes until protected load is actually at risk. Exercise the whole arrangement with regular fault-injection drills — deliberately make a node hot and confirm that the slowdown stays contained, that protected traffic keeps flowing, and that the system ramps back by itself within minutes. Keep a manual drain switch as a last-resort escalation only; if operators ever have to use it, treat that as a defect in the automatic loop.

Final operating rule: every caller throttles itself under failure instead of amplifying it; every dependency is isolated behind its own breaker and bounded pool so a local fault stays local; under pressure the system sheds by priority with headroom and hysteresis rather than collapsing; and it returns to full service through a staged, self-driven, automatically reversible ramp — so a hot spot degrades a slice, never the whole, and heals without a human pulling traffic.
===RECOMMENDATION_END===
