# Working (discarded before grading)

## State

**problem.** Intermittent total outages. A modest demand spike slows a few backend nodes; clients/internal callers auto-retry failed requests, multiplying load on already-struggling nodes, pushing more nodes over the edge until the whole service saturates. Recovery requires manual traffic-shedding. Team has good monitoring, can change client/server behavior, set limits, add capacity, but cannot indefinitely over-provision.

**meta_model.** A small local perturbation feeds a *reinforcing* feedback loop (the corrective response — retrying — is itself the amplifier), which propagates across tightly coupled units past a tipping threshold into total collapse, with no balancing loop to pull the system back, so it cannot self-recover.

## Verbs run (in order, with one-line why)

1. **match** (search_prime) — anchor the meta-model. Surfaced: circular_causality, homeostasis, instability, cascade, threshold_and_criticality, perturbation, resonance/amplification, self_organization.
2. **match** (search_by_facets) — oblique, multi-structure problem; decompose into 7 facets (amplifying loop / chain-reaction spread / response-increases-load / no-slack coupling / no self-recovery past threshold / rate-limiting a shared resource / isolating a failing part). Per-facet surfaced system_archetypes, systemic_risk, coupling, load_balancing, controlled_reentry, robustness, functional_redundancy, plus stress_rupture/regime_change at the fusion top.
3. **salience-rank** — load-bearing core: (a) circular_causality — the retry loop is a *positive* loop where the response IS the amplifier; (b) homeostasis — the missing balancing loop + envelope limits (T2); (c) controlled_reentry — self-recovery must be staged, not a toggle (T1, T4); (d) coupling/cascade — propagation across nodes; threshold the tipping point.
4. **prune** — drop resonance/synchronization/self_organization (descriptive, not actionable here); keep the four operative primes + the cascade/threshold framing.
5. **get_prime** circular_causality — confirms: positive loop → exponential blow-up; intervene at *loop structure/leverage points*, not the visible hot node; cites retry-storm/circuit-breaker as the canonical loop-interruption.
6. **get_prime** homeostasis — confirms the cure is a *negative* (balancing) loop: sensor→comparator→actuator with finite correction authority and response time; envelope must reject the disturbance class; T5 warns of overshoot/oscillation (over-throttling = the failure criterion).
7. **get_prime** controlled_reentry — confirms recovery is asymmetric to failure: staged ramp with go/no-go gates and re-suspend capability; T4 warns reentry can re-create the original overload (so ramp must back off on the same signal that tripped it).
8. **match** (search_archetype) — concrete mechanism set: cycle_breaking, recovery_interval_design, coordination_across_reentry_phases, intermittent_burst_absorption, **cycle_staggering** (de-synchronize retries — the jitter mechanism), tapering_strategy, elastic_capacity_scaling.
9. **compose / lower** — assemble into the typed model and instantiate concrete mechanisms, rules, thresholds, paths, and the operating rule below.
10. **evaluate-fit (gate)** — check against the explicit failure mode (over-aggressive throttling routinely degrading normal traffic). Resolved by load-shedding by *priority class* + adaptive limits keyed to live latency, not static caps; and by hysteresis/jittered ramp to prevent oscillation (homeostasis T5).

**model (typed).** Nodes: clients/callers, edge/admission layer, backend node pool, shared dependency. Edges: request→node (load), node-slow→client (failure signal), client→node (RETRY = the reinforcing edge to cut), operator→system (manual shed = the missing balancing edge to automate). Leverage point = the retry edge + an admission/balancing loop, NOT the hot node.

COST: ~9 operations, ~22k tokens.

===RECOMMENDATION_BEGIN===
The outages are not a capacity problem; they are a feedback problem. A small slowdown is being amplified by the very thing meant to cope with it — automatic retries — turning a local dip into a self-reinforcing loop that spreads across nodes until everything saturates. The fix is to cut the amplifier, give every overloaded part a way to protect itself, and build in an automatic, gradual return to normal so no human has to shed traffic.

**Cut the amplifier.** Retries are the engine of collapse, so make them cheap and self-limiting. Cap retries to at most one attempt, and only retry errors that are actually retryable. Gate every retry behind a token-bucket retry budget per client: if more than roughly 10% of recent requests are already retries, stop retrying and fail fast. Add full jitter to all backoff so that callers who failed together do not return in a synchronized wave. Critically, a node returning an overload signal must tell callers *not* to retry it (a "retry-after / do-not-retry" response), converting a retry storm into immediate, quiet shedding.

**Let each node protect itself.** Give every server an admission controller that watches its own live latency and queue depth, not just CPU. When in-flight concurrency or queue latency crosses a threshold, the node immediately rejects excess work with a fast, cheap overload response rather than slowly grinding through it — shedding load is what keeps it from tipping. Bound every queue; an unbounded queue just hides the collapse. Shed by priority class: health checks and high-value user traffic are admitted last to be dropped, bulk and background work first, so protection never degrades normal interactive traffic until genuinely necessary.

**Stop failure from spreading.** Put a circuit breaker on every call between services and to shared dependencies. When error or latency from a downstream target crosses a threshold, the breaker opens and callers fail fast (serving degraded or cached responses) instead of piling load onto a struggling target. This isolates a hot spot so it cannot drag its callers down with it. Apply concurrency limits per downstream dependency so one slow dependency cannot consume all of a caller's worker threads.

**Make recovery automatic and gradual.** Recovery must not be a single switch back to full load — that simply re-creates the overload that caused the outage. After a breaker opens, it moves to a half-open state and admits only a trickle of probe traffic. Real traffic is then ramped back in stages (for example, 1% → 10% → 50% → 100%), and at each stage the controller watches the same latency and error signals that triggered the trip. If they stay healthy for a short hold period, it advances; if they degrade, it steps back down. Jitter and stagger these ramp-ups across nodes and clients so recovery itself does not arrive as a synchronized surge. This is the balancing loop the system currently lacks: it senses load, corrects, and walks itself back to normal within minutes without an operator.

**Keep the safeguards from over-correcting.** The danger is throttling so hard that normal traffic is routinely degraded. Prevent this by keying limits to live measured latency rather than fixed caps, so headroom expands automatically when the system is healthy; by using hysteresis (trip quickly, recover only after a sustained healthy interval) so the system does not flap between open and closed; and by tuning thresholds so that under normal load the breakers stay closed and admission controllers reject essentially nothing. Use any spare capacity and fast autoscaling as a buffer for genuine spikes, but treat shedding and breaking — not over-provisioning — as the primary defense, since hardware cannot be scaled without limit.

**Final operating rule.** Each component continuously measures its own load and the health of everything it depends on. When load exceeds what it can serve within target latency, it sheds or fails fast immediately — lowest-priority work first — rather than absorbing the overload; when a dependency is unhealthy, it stops sending it work. Recovery is never a single jump back to full load: traffic is re-admitted in jittered stages, advancing only while the health signals stay green and stepping back the moment they do not. A local slowdown thus stays local, and the system returns itself to full service without anyone manually shedding traffic.
===RECOMMENDATION_END===
