# Working (control policy: free verb order; discarded before grading)

STATE.problem: intermittent total outages; a modest demand spike slows a few backend nodes; automatic client/internal retries multiply load on the struggling nodes, pushing more over the edge until full saturation; recovery needs manual traffic-shed. Need: keep local slowdown local AND self-recover within minutes, without routinely degrading normal traffic.

VERBS RUN (in order, with one-line why):
1. **match** — search_prime with the domain-stripped meta-model ("local perturbation triggers an amplifying feedback response that escalates to total collapse; cannot self-recover"). Surfaced the amplifying-loop/collapse neighborhood.
2. **match (search_by_facets)** — six structural facets (amplifying-overload loop; contagious local→total failure; uncoordinated demand exhausting a shared resource; retries multiplying load; cannot self-reset; rate-limit to protect downstream). Per-facet lists pulled the operative spread: cascade/systemic_risk, tragedy_of_the_commons/scarcity, buffering/bottleneck/load_balancing, controlled_reentry/resilience/fault_tolerance, hysteresis.
3. **salience-rank** — load-bearing core = the re-emission/amplification loop (cascade) + the retry-incentive misalignment (tragedy_of_the_commons); recovery half = controlled_reentry + hysteresis. Periphery (instability/perturbation/tipping_points) = vocabulary, pruned.
4. **prune** — cut to: cascade, tragedy_of_the_commons, controlled_reentry, hysteresis (+ archetypes instability_dampening, balancing_loop_stabilization).
5. **lift** — abstract structure: each node re-emits the disturbance (retry = re-emission); criticality parameter k = amplification factor per failed request. Containment menu (from cascade): cut coupling edges, raise neighbor thresholds, insert firebreaks that absorb re-emission before it regenerates. Keep k<1.
6. **transport (+map)** — instability_dampening archetype: map "where is the gain?" → the retry loop is the gain. Its mechanisms (rate limiter, exponential backoff, bulkhead/segment isolation, circuit breaker, staged-escalation gate) align 1:1 to the target. Roles: perturbation=demand spike; amplification path=retries×fan-out; firebreak=load-shed/circuit break.
7. **compose** — typed model: clients/internal-callers --(retry edge, the amplifier)--> backend nodes --(shared dependency edges)--> downstream pools. Annotate: retry edge = tragedy_of_commons (private-rational retry, collective-destructive); node→pool edges = cascade coupling; recovery path = controlled_reentry; trip-vs-recover asymmetry = hysteresis.
8. **lower** — instantiate concrete mechanisms/thresholds (below).
9. **reconcile** — recovery loop tuned via balancing_loop_stabilization: separate activation/deactivation thresholds (hysteresis band) to prevent flapping; deadband + effect-monitoring so it does not over-damp normal traffic.
10. **evaluate-fit (gate)** — checked instability_dampening anti-signatures: NOT mere capacity absence (it is genuine amplification) ✓; NOT a case where amplification is wanted ✓. Failure modes guarded: over-damping (→ the failure criterion "routinely degraded normal traffic": use deadband, prioritized shedding, fast recovery); displaced instability (→ bulkheads per dependency, not one global throttle); hidden accumulation (→ alert on sustained shed/breaker-open as a redesign signal). Gate PASSED.

COST: ~9 operations, ~24k tokens.

===RECOMMENDATION_BEGIN===
The outage is not a capacity problem; it is an amplification problem. A small slowdown is harmless on its own — what makes it total is that every failed or slow request is answered with more requests, so the system's own response to trouble becomes the fuel for more trouble. Each retry is individually reasonable but collectively destructive: a caller that retries gets a marginally better chance for itself while adding load that everyone, including itself, pays for. The whole design below has one goal: make the load added per unit of trouble less than one, so disturbances shrink instead of growing, and build a recovery path that the system walks by itself.

**Cap the amplifier at the source (retries).** Give every client and internal caller a *retry budget*: retries may be at most a small fraction (start at 10%) of that caller's recent successful requests, enforced over a rolling window. When the budget is exhausted, failures are surfaced immediately rather than retried. Retries must use exponential backoff with full random jitter (so callers do not re-converge into synchronized waves) and must stop entirely once a response carries an explicit "overloaded, do not retry" signal. Forbid nested retries: a request that is itself a retry of an upstream call is never retried again, so a single user action cannot fan out into a storm.

**Make overloaded servers reject fast instead of dying slow.** Each node advertises load via response headers. When a node's queue depth or latency crosses a high-water mark, it sheds *new* load by rejecting requests quickly with a "retry-after" hint and the no-retry signal, while protecting in-flight work — a fast cheap rejection cannot itself cascade, but a slow timeout can. Shed lowest-priority traffic first (background, best-effort, then bulk) so essential traffic keeps flowing; this is what keeps normal service from being broadly degraded.

**Cut the coupling so a hot spot stays a hot spot.** Partition callers and dependencies into isolated pools (bulkheads) with separate connection/concurrency limits, so one saturated dependency cannot consume the threads or connections that healthy paths need. Put a *circuit breaker* in front of each dependency: when a callee's error or timeout rate over a short window exceeds a threshold (e.g. 50% over 10 seconds), the breaker opens and calls fail fast locally for a cooldown, removing load from the struggling node entirely. The breaker is the firebreak — it stops re-emission rather than merely slowing it.

**Recover automatically, with a deliberate gap between tripping and re-arming.** Recovery is not the reverse of the failure; it is a separately engineered ramp. After cooldown the breaker goes *half-open* and admits a trickle of probe traffic; it closes only after sustained success, and re-opens on the first renewed failures. Use *different thresholds to shed and to recover* than the levels at which you stop shedding — recover at a comfortably lower load than the level that triggered shedding (e.g. trip at 80% utilization, resume at 60%). This gap prevents the system from flapping on and off at the boundary. When a node recovers, ramp its admitted traffic back up in stages (e.g. 10% → 25% → 50% → 100%), advancing only while latency and error rates stay healthy and stepping back automatically if they degrade, so a node that is not truly well is not slammed back to full load.

**Keep it from quietly over-protecting.** Add a deadband so ordinary variation triggers nothing, and treat *frequent or sustained* shedding, open breakers, or exhausted retry budgets as alarms — not as normal operation — because that signature means real capacity is short and the safeguards are now masking it. Add capacity or raise limits in response to that signal, not to transient spikes.

**Operating rule:** every component, at every moment, must add less load in response to trouble than the trouble it is responding to — cap retries, reject fast, isolate and break the coupling at defined load thresholds, and re-admit traffic only gradually and only once health is sustained, with a clear gap between the level that triggers protection and the lower level that releases it. If a localized slowdown can still recruit the rest of the system into amplifying it, the loop is not yet closed.
===RECOMMENDATION_END===
