# c22 — runaway-retry cascade / self-recovery design

## STATE

**problem.** Intermittent total outages. A modest demand spike slows a few backend nodes; automatic client/internal retries multiply load on the struggling nodes; more nodes tip over; the service saturates. Recovery needs manual traffic-shedding. Levers available: client + server behavior, limits, capacity, good monitoring; cannot indefinitely over-provision.
- success: a local slowdown stays local AND the system self-recovers within minutes without manual shedding.
- failure: a single hot spot can still cascade to total outage, OR safeguards throttle so hard that normal traffic is routinely degraded.

**meta_model (L1, domain-stripped).** Roles: a perturbation source (small load increase); a population of coupled processing units sharing capacity; agents that react to slow/failed responses by re-issuing demand; a shared exhaustible resource. Relations: a *reinforcing* loop (failure → retry → more load → more failure) overwhelms the system's weak *balancing* loops; tight coupling lets local failure propagate; no boundary isolates a failing unit; the system has no autonomous return path to a viable range (recovery is exogenous/manual). Invariants to hold: bound the amplification of small disturbances; keep failure local; provide an autonomous balancing loop that returns to a viable operating range; preserve responsiveness to genuine demand.

**operative_primes.**
- reinforcing-loop amplification / instability — *load_bearing* (the retry storm IS the disease).
- propagation / cascade across coupled units (systemic_risk, no isolation) — *load_bearing*.
- ultra-stability / homeostasis — *load_bearing* (autonomous return to a viable *range*, not a fixed point; requisite variety of regulatory responses).
- interference_and_contention on a shared resource — *context* (the substrate the loop runs on).
- hysteresis (asymmetric leave/return thresholds) — *load_bearing for the recovery half*.

**candidates (verbs: match via search_by_facets + search_archetype).**
- `instability_dampening` (overlap: instability, damping, feedback; related: amplification, threshold, homeostasis, resilience) — central; literally names retry-storms, backoff, circuit-breaker, bulkhead isolation, safe-fallback-with-reentry.
- `balancing_loop_stabilization` (circular_causality, feedback, homeostasis) — the autonomous self-correction loop with anti-overshoot tuning.
- supporting recognition only: `hysteresis_management` (separate trip/re-admit thresholds), `boundary_permeability_control` (the isolating boundary), `equilibrium_restoration` (self-recovery target).

**VERBS RUN (history).**
1. match (search_by_facets, 5 facets) — oblique multi-structure problem; surfaced reinforcing-loop, cascade/systemic_risk, ultra-stability, contention.
2. match (search_archetype) — `instability_dampening` 0.79 top.
3. get_archetype ×2 + get_prime — pulled full component lists + ultra-stability signature.
4. match (search_archetype, isolation/recovery facet) — confirmed hysteresis + boundary + equilibrium-restoration as recognition, not new structure.
5. compose — assembled the reinforcing/balancing/boundary model (referential integrity: every operative prime annotated).
6. lift (L2 coverage check, my choice — yes) over `instability_dampening` + `balancing_loop_stabilization` components.

**coverage_map (L2 — every component statused).**
- Amplification-pathway map → *instantiated*: retry-on-failure is the gain path; attack it directly.
- Gain / cadence control → *instantiated*: bounded retries + exponential backoff with jitter + a per-caller retry budget.
- Propagation boundary → *instantiated*: bulkheads (per-dependency concurrency pools) + per-node circuit breakers.
- Buffer / absorptive capacity → *instantiated*: bounded queues + fast load-shedding (reject early, cheaply).
- Balancing loop / correction rule → *instantiated*: adaptive concurrency limit driven by latency/error deviation; autoscaling as slow loop.
- Hysteresis / reentry → *instantiated*: breaker half-open probe; separate trip vs. re-admit thresholds; gradual ramp.
- Safe fallback state → *instantiated*: degraded-but-up responses (cached/partial/defaults), not failure.
- Over-damping guard (the over-lift / over-throttle guard) → *instantiated*: prioritize by criticality, size limits to spare capacity, idempotency keys, observe both cascade AND false-throttle.
- Saturation / escalation → *correctly_declined as a primary lever*: manual escalation kept only as last-resort floor; cannot over-provision, so not the design.
- Residual-instability monitor → *instantiated*: track shed rate + breaker state + retry-budget exhaustion.
No component `missing` → STOP permitted.

COST: ~6 operations (5 MCP calls + 1 write), ~14k tokens.

===RECOMMENDATION_BEGIN===
The outages are not caused by the demand spike itself but by how the system reacts to it: a failed or slow response triggers automatic retries, retries pile new load onto the nodes that are already struggling, and that extra load pushes more nodes over the edge. The fix is to cut the strength of that self-feeding loop, stop a local failure from spreading, and give the system a built-in way to settle back to normal on its own.

Start at the retries, because that is where small problems get multiplied. Every client and internal caller should retry only failures that are genuinely worth retrying, cap retries at one or two attempts, and space attempts using exponential backoff with random jitter so callers do not resynchronize into a fresh wave. Most importantly, give each caller a retry budget: allow retries to consume at most a small fraction (for example 10%) of its request volume over a rolling window, and when that budget is exhausted, stop retrying and fail fast. A retry budget is what structurally prevents a retry storm, because total load can no longer balloon to a multiple of real demand.

Next, keep a hot spot from becoming a system-wide fire by adding boundaries that the failure cannot cross. Give each downstream dependency its own bounded pool of concurrent calls (a bulkhead) so that one slow dependency cannot consume all the threads, connections, or memory and starve unrelated work. In front of each dependency or node, run a circuit breaker: track its recent error and latency rate, and when that crosses a trip threshold, open the breaker and immediately reject calls to it for a short cool-off instead of queuing them. Rejecting cheaply and instantly is what protects the struggling node and lets it drain its backlog and recover.

Make the system shed the right load rather than collapse under all of it. Put a bounded queue with admission control in front of each node; when the queue is near full or latency climbs, reject the lowest-priority excess at the edge before it consumes capacity, returning a clear "busy, retry later" signal (which, combined with the retry budget, will not rebound into a storm). Classify traffic by importance so that under pressure the system preserves critical requests and sheds optional ones first. This is also the safeguard against the opposite failure: because shedding is prioritized and sized to protect spare capacity, normal traffic is not routinely degraded — only genuine overload is touched.

To remove the need for manual recovery, close a self-correcting loop on each node. Continuously measure each node's latency and error rate against a healthy target band, and let an adaptive concurrency limiter raise the allowed in-flight requests when the node is healthy and lower it when latency or errors rise — an automatic, fast-acting throttle that finds the safe load level on its own. Use asymmetric thresholds (hysteresis): trip protections quickly when health degrades, but require sustained good health before fully re-admitting traffic, so the system does not flap between open and closed. The circuit breaker enforces this directly: after its cool-off it goes half-open, lets a trickle of probe requests through, and only fully reopens if those succeed, then ramps traffic back gradually rather than all at once. Pair this with autoscaling as a slower loop that adds capacity when sustained healthy demand is high, while the fast loops hold the line in the seconds before new capacity arrives.

Finally, prefer degrading over failing. When a dependency's breaker is open, return a cached, partial, or default response where the request allows it, so a local outage shows up as reduced richness rather than a hard error that provokes more retries. Make write operations idempotent (with request keys) so the bounded retries that remain are safe. Keep monitoring on the new safeguards themselves — shed rate, breaker open/close events, retry-budget exhaustion, and the adaptive limits — so you can see both kinds of trouble: a cascade leaking through, or throttling that is biting healthy traffic.

Operating rule: every caller retries sparingly within a strict retry budget; every dependency runs behind a bulkhead and a circuit breaker that trips fast and re-admits slowly via half-open probing; each node self-tunes its accepted load to a healthy latency/error band and sheds prioritized excess early; capacity scales on the slow loop. A slowdown then stays contained to the affected component, and the system walks itself back to full service in minutes — with manual traffic-shedding kept only as a never-needed last resort.
===RECOMMENDATION_END===
