# c13 — Cascading-overload / retry-storm resilience design

## Working (discarded before grading)

**Step 1 — Problem statement.** Decision: design mechanisms so a localized backend slowdown cannot escalate into full-service saturation, and so the system self-recovers without manual traffic-shedding. Bounds: can change client+server behavior, set limits, add capacity, but cannot indefinitely over-provision; have good monitoring. Success: a hot spot stays local AND service returns to normal within minutes with no manual intervention. Failure: a single hot spot can still cascade to total outage, OR safeguards throttle so hard that normal traffic is routinely degraded. The core dynamic is a reinforcing loop: slow node → failed/slow request → automatic retry → multiplied load → more slow nodes → saturation.

**Step 2 — Operative primes (search_prime / facets; err toward inclusion).** Ran `corpus_stats` (643 primes/625 archetypes), one `search_by_facets` (6 facets), three `search_prime` queries with domain-stripped meta-models. Surfaced: circuit_breaker, controlled_reentry, cascade, feedback, circular_causality, instability, threshold_and_criticality, tipping_points, homeostasis, saturation (receptor_saturation), buffering, system_slack, margin_of_safety, load_balancing, scalability, tragedy_of_the_commons, systemic_risk, adaptive_capacity, resilience. Pulled full records for circuit_breaker and controlled_reentry.

**Step 3 — Salience rank.** (1) feedback/circular_causality — the amplifying retry loop is the engine of the whole failure. (2) circuit_breaker — the direct structural answer: trip-isolate-hold-reset. (3) cascade / systemic_risk — names the propagation to be stopped. (4) saturation / margin_of_safety / buffering — capacity headroom and bounded queues. (5) controlled_reentry — self-recovery without manual shedding. (6) instability / threshold — the gain/criticality to tune. Lower: load_balancing, scalability (relevant but not load-bearing for cascade arrest), tragedy_of_the_commons (retries as a commons, oblique).

**Step 4 — Operative subset (~6).** feedback (the gain to kill), circuit_breaker (fast isolation), cascade/systemic_risk (propagation boundary), saturation+buffering+margin_of_safety (admission + headroom), controlled_reentry (automatic recovery), instability/threshold (tuning). Pruned load_balancing and scalability (helpful background, not the cascade-arrest mechanism); pruned tragedy_of_the_commons (metaphor only).

**Step 5 — Context model (typed relational structure).**
- Nodes: `client/internal-caller` (retry logic), `entry-gateway/admission`, `backend-node-pool` (partitioned), `dependency-edge` (caller→callee), `monitor/control-loop`.
- Edges: caller --issues_request--> backend; backend --slows_when--> overloaded; caller --auto_retries(on fail/timeout)--> backend [AMPLIFY: feedback]; backend --spreads_load--> neighbor backend [cascade]; monitor --observes--> error_rate/latency; control --trips/throttles/resets--> edges & admission.
- Prime annotations: the retry edge carries `feedback` (positive gain); the caller→callee edge is where `circuit_breaker` belongs; pool partitioning is the `cascade`/`systemic_risk` propagation boundary; admission carries `saturation`/`margin_of_safety`; recovery carries `controlled_reentry`.

**Step 6 — Meta-model (domain stripped).** A network of dependent processors under variable demand contains a reinforcing loop: when a processor degrades, dependents react in a way that increases load on the degraded element, which degrades neighbors via shared coupling until the whole reaches a tipping point. The fix is to (a) cut the loop gain, (b) install threshold-triggered protective interrupters that isolate a degrading element before it propagates, (c) bound admitted demand against finite capacity with protective headroom, and (d) make re-engagement staged and self-testing so the system returns to normal without external action — all while keeping false-trips low enough that ordinary variation is not suppressed.

**Step 7 — Archetype query (find_archetypes_for_prime, search_archetype).** circuit_breaker and cascade have zero catalogued archetypes (find_archetypes_for_prime = 0). `search_archetype` on the meta-model returned, and I pulled full records for: **instability_dampening** (find the gain, damp it, preserve true signal), **saturation_avoidance** (protect a bounded channel; admission limits, priority, degraded mode), **intermittent_burst_absorption** (detect spike, absorb, prioritize, recover), **elastic_capacity_scaling** (governed capacity loop with cooldown/hysteresis).

**Step 8 — Reason via both views.** Context view: the retry edge is the gain (instability_dampening → backoff + retry budget); the caller→callee edge gets a circuit_breaker (open on error/latency threshold, half-open probe = controlled_reentry); admission gets saturation_avoidance (load-shed lowest priority, keep degraded service alive — note circuit_breaker T5: prefer degradation over hard amputation); pool partitioning = bulkheads (cascade boundary). Meta-model view confirms: kill gain + threshold-interrupt + bounded admission + self-testing reentry. elastic_capacity_scaling supplies bounded burst headroom (respecting "cannot indefinitely over-provision").

**Step 9 — Fit / anti-signatures.** instability_dampening anti-sig "disturbance already large, needs recovery not sensitivity-reduction" → partly trips, so I pair damping with circuit_breaker + reentry (recovery), not damping alone. saturation_avoidance anti-sig "decisive move is dropping work after overload" → that is load-shedding, which I include as the degraded-mode tier, so fine. circuit_breaker T1 (false vs missed trips), T3 (reset embeds fault theory), T5 (can convert tolerable degradation to hard outage) → mitigate via hysteresis, half-open probes, and prefer shed-then-break ordering. elastic_capacity_scaling anti-sig "capacity can't change in horizon" → autoscale lags a fast retry storm, so it's secondary headroom, not the primary guard. No candidate is fully tripped; all four contribute. Final design below.

COST: ~10 operations, ~46k tokens

===RECOMMENDATION_BEGIN===
The outages are driven by a self-reinforcing loop, not by raw demand: a few slow nodes cause timeouts, callers retry automatically, the retries multiply load on exactly the nodes least able to absorb it, and the overload spreads through shared dependencies until everything saturates. The design must therefore do four things at once — cut the loop's gain, isolate trouble before it spreads, bound how much work the system admits, and make recovery automatic — without throttling so hard that ordinary traffic suffers.

First, defang retries. Every client and internal caller should retry with exponential backoff plus random jitter, never on a fixed immediate timer, and each caller should hold a retry budget: retries may consume only a small fraction (for example 10 percent) of that caller's outbound requests over a rolling window. When the budget is exhausted, further failures fail fast rather than being retried. This single change removes most of the amplification, because it prevents a dip in success rate from being converted into a multiplied wave of new load.

Second, put a tripping interrupter on each caller-to-dependency path. Each caller tracks the recent error and latency rate to a given backend. When that rate crosses a danger threshold, the path "opens": calls fail fast (returning a cached value, a default, or a clear error) instead of piling onto a dying dependency and exhausting connection pools. Crucially, before fully opening, prefer to keep degraded-but-working service flowing — open only when continuing would clearly make things worse — so the interrupter does not turn a tolerable slowdown into a self-inflicted outage. Set the open threshold with hysteresis (a gap between the level that trips and the level that recovers) so the path does not flap rapidly between states.

Third, isolate by partition. Group backend nodes and the resources they share (connection pools, thread pools, queues) into bulkheads so that a hot spot consuming one partition's capacity cannot drain the capacity the rest of the system needs. A fault then stays confined to its compartment instead of propagating across the whole pool.

Fourth, bound admitted work against finite capacity. At the entry point and at each saturating service, enforce admission control: queues are bounded, and when a queue or concurrency limit is reached, the lowest-priority work is shed first (and shed early, before collapse), while high-priority and already-in-flight work is protected. Operate each service with deliberate headroom — a safety margin below its theoretical maximum — and treat that margin as the normal operating ceiling, not a number to be eliminated for efficiency. Bursts that fit inside the margin are absorbed; bursts that exceed it are shed at the edge, where shedding is cheap, rather than deep in the system where it cascades. Autoscaling can add capacity as a secondary cushion, but because it lags a fast retry storm it must not be the primary guard.

Fifth, make recovery self-testing. Recovery must not require an operator to manually shed traffic. After an interrupter opens, it waits a cooldown and then enters a trial state: it lets a single probe request (or a trickle) through. If the probe succeeds, it ramps traffic back gradually — not all at once — watching error and latency the whole way; if the probe fails, it re-opens and waits again. This staged, monitored re-entry is what lets a localized slowdown clear on its own within minutes: the loop gain is already gone, the fault was confined, and the system gently re-tests the path until it is genuinely healthy, ramping rather than slamming load back on so it does not immediately re-trip.

Tuning is where this succeeds or fails. The thresholds for tripping, shedding, and backoff must be calibrated so that ordinary variation — normal latency jitter, a routine traffic bump — does not trip them, while a genuine degradation does. Use the existing monitoring to drive this: trip on sustained error/latency excursions, not single spikes; require a signal to persist briefly before acting; and review false trips and missed trips regularly, retuning as load patterns change. Keep every control observable, so operators can see why traffic was slowed, capped, or interrupted, and can override in a true emergency.

The final operating rule: never let a failed request generate more total work than the original — degrade, shed the least important load early, and break a connection before retries can multiply it; confine every fault to its own compartment; run with deliberate headroom rather than at the edge; and recover by quietly probing and ramping a path back to health, never by slamming full load onto a node that has not yet proven it is well.
===RECOMMENDATION_END===
