# c28 — Cascading-overload resilience design

## Working (discarded before grading)

**Step 1 — Problem statement.** Decision: design mechanisms that (a) stop a localized backend slowdown from escalating into total saturation and (b) let the system self-recover without manual traffic-shedding. Bounds: can change client + server behavior, set limits, add capacity, good monitoring; cannot indefinitely over-provision. Success: a hot spot stays local AND service returns to normal within minutes without operators dumping traffic. Failure: a single hot spot still cascades to total outage, OR safeguards are so aggressive that normal traffic is routinely degraded. The control problem is therefore two-sided — suppress amplification without suppressing legitimate load.

**Step 2 — Operative primes (search_prime + search_by_facets, err toward inclusion).** Ran a blended semantic query and an 8-facet decomposition. The facet view was decisively better: it surfaced the load-bearing structural primes that the blended query buried. Identified: `cascade` (local trigger → coupled re-emission → self-perpetuating propagation), `circuit_breaker` (monitor → trip → isolate → held-open → reset), `interference_and_contention` / `scarcity` (shared saturated resource), `instability`/`ultra_stability_ashby_s_concept` (cannot self-return after perturbation; maintain essential variables in bounds), `controlled_reentry` (staged, monitored return after suspension — the recovery half), `load_balancing`, `caching`, `threshold_and_criticality`, `concurrency`/`randomness` (desynchronize correlated retries), `bottleneck`, `prioritization`, `modularity_and_hierarchical_decomposability` (bulkheads). Retrieved full records for cascade, circuit_breaker, ultra_stability, controlled_reentry.

**Step 3 — Salience rank.** (1) `cascade` — names the exact failure: retries re-emit load, each stressed node becomes a new source; k>1 super-critical. (2) `circuit_breaker` — the decisive cut-re-emission lever with a built-in self-reset (half-open). (3) `interference_and_contention`/`scarcity` — the shared resource being saturated. (4) `controlled_reentry` — the self-recovery half, staged ramp not a flip. (5) `ultra_stability` — keep essential variables (latency, in-flight count) in bounds via second-order regulation. (6) `concurrency`/`randomness` — jittered backoff to kill synchronized retry storms. (7) `load_balancing`/`prioritization`/`caching` — supporting load-relief. Lower: temporal_decay, scaling (over-provisioning is barred).

**Step 4 — Prune (operative subset, 7).** cascade; circuit_breaker; interference_and_contention; controlled_reentry; concurrency/randomness; load-shedding-by-prioritization; ultra_stability. Rationale: each maps to one mechanism slot. Dropped antifragility/tolerance/stressor-adaptation (semantic neighbors, not load-bearing here) and pure elastic scaling as a *primary* fix (cannot over-provision indefinitely — but admissible as a bounded secondary).

**Step 5 — Context model (typed graph).**
Nodes: Client, InternalCaller, LoadBalancer, BackendNode[i], SharedResource(pool/DB), Monitor.
Edges: Client/InternalCaller --request--> LB --route--> Node[i] --contends-for--> SharedResource; Node[i] --slow→timeout--> Client; Client --retry(amplifies)--> LB [CASCADE re-emission edge]; Node[i] --redistributes-load--> Node[j] [coupling edge]; Monitor --observes--> {latency, in-flight, error-rate}. The retry edge is the re-emission that makes k>1. Annotated: contention=interference; retry-loop=cascade; LB=load_balancing; Monitor+thresholds=ultra_stability second-order regulator.

**Step 6 — Meta-model.** A network of thresholded elements sharing a finite resource; a local threshold crossing triggers compensating re-emission (retry) that raises neighbors' load past their thresholds → super-critical propagation. Goal: hold the system sub-critical (k<1) by raising thresholds, cutting coupling/re-emission, and absorbing bursts; then stage a monitored return. Essential variables (in-flight requests, queue depth) must stay in bounds.

**Step 7 — Archetype query (search_archetype + get_archetype).** Top fused hits: `fault_tolerant_operation`, `instability_dampening`, `intermittent_burst_absorption`, `elastic_capacity_scaling`, `cycle_staggering`, `tipping_point_prevention`. Pulled full records for the first four. `instability_dampening` is the closest single parent (where is the gain? — the retry loop; levers: gain reduction, buffers, cadence control, propagation boundaries, circuit-breaker). `fault_tolerant_operation` supplies isolation-boundary + continuation-mode + recovery framing. `intermittent_burst_absorption` supplies bounded queue + shed-gate + recovery-interval. `controlled_reentry` (prime) supplies the staged self-recovery.

**Step 8 — Reason via both views.** Context view: cut the re-emission edge (cap + jittered backoff + per-dependency breakers), firewall coupling (bulkhead pools), absorb (bounded queues), shed lowest-priority at admission, and stage automatic reentry (half-open probe + ramp). Meta view: every move either lowers k (cut coupling, cap retries), raises thresholds (concurrency limits with headroom), or absorbs the burst — and the reset is a probe, not a flip. The two views agree.

**Step 9 — Anti-signature / trigger checks.** `elastic_capacity_scaling` — TRIPS the over-provision bound as a *primary* cure; keep only as bounded autoscale secondary. `fault_tolerant_operation` anti-sig "use fail-safe if fault can't be bounded" — does not apply; faults are bounded and continuation is wanted, so don't hard-stop everything. `circuit_breaker` T5 (can convert tolerable degradation into hard outage) and `instability_dampening` over-damping — directly drives Failure criterion 2; mitigated by per-dependency scope + hysteresis + priority-preserving shed so normal traffic isn't routinely degraded. `intermittent_burst_absorption` anti-sig "sustained load → redesign baseline" — if peaks become permanent, raise baseline (out of scope but flagged). All survivors reconciled into the recommendation below.

COST: ~13 operations, ~46000 tokens

===RECOMMENDATION_BEGIN===
The outages are not caused by the size of the demand spike; they are caused by how the system repays it. A few slow nodes time out, callers retry, the retries pile onto the already-slow nodes, more nodes tip over, and the disturbance regenerates itself until everything is saturated. The cure is to make the system unable to amplify a local slowdown, and to make its return automatic. Treat in-flight request count and queue depth as the vital signs you defend, and keep them inside fixed bounds at every node.

First, stop the retry storm, which is the engine of the cascade. Give every client and internal caller a strict, small retry budget — at most one retry, and only for requests that are safe to repeat — governed by a token bucket so that retries can never exceed a few percent of fresh traffic. Space retries with exponential backoff plus randomized jitter so that thousands of callers do not resynchronize into a second wave. A request that has already waited past its deadline must be abandoned, not retried: enforce deadline propagation so a caller never works on a response no one is waiting for.

Second, cut the coupling so one hot spot cannot recruit its neighbors. Give each backend node and each downstream dependency a bounded concurrency limit — a ceiling on simultaneous in-flight requests sized to leave headroom below the saturation point. When the ceiling is reached, the node sheds the excess immediately rather than queuing it indefinitely; queues must be short and bounded so latency cannot grow without limit. Isolate pools per dependency (separate connection and thread pools for each backend or database) so that a single slow dependency exhausts only its own pool and healthy paths keep flowing.

Third, fail fast on dependencies that are actually sick. Wrap each call to a backend or shared resource in a monitored cut-off: when its error rate or latency crosses a threshold over a short rolling window, the path opens and callers get an immediate error or a cached or degraded fallback instead of stacking up on a dying dependency. Scope each cut-off narrowly to one dependency, never globally, and add a gap between the trip threshold and the recovery threshold so it cannot rapidly flap between open and closed.

Fourth, when load still exceeds safe capacity, shed by priority rather than collapsing. At admission, classify traffic — health checks and high-value user requests first, retried and low-value background work first to be dropped. Under pressure the system rejects the least important load early and cheaply, protecting the throughput that matters. This is what keeps a hot spot localized: excess is refused at the edge instead of propagating inward.

Fifth, make recovery a staged, automatic process, not a manual flip. A cut-off path does not slam back to full traffic; after a cooldown it lets a single trial request through, and only on success does it ramp load back in stages — a small fraction, then more, watching error rate and latency at each step and stepping back down if the vital signs deteriorate. Concurrency limits and admission thresholds relax along the same gradient. Because shedding and cut-offs relieve the overloaded nodes the instant they trip, the nodes drain their backlogs within seconds, the trial probes succeed, and the ramp restores full service in minutes with no operator dumping traffic.

Tune against both failure modes at once. Set concurrency ceilings and trip thresholds from observed healthy latency with deliberate headroom, so they fire on genuine overload but tolerate ordinary variation — otherwise the safeguards themselves become the outage by throttling normal traffic. Keep the hysteresis gap and backoff jitter wide enough to prevent oscillation. Bounded autoscaling may add capacity for sustained higher load, but it is a slow secondary measure, never the primary defense, and never a substitute for the limits above; if peaks become permanent, raise the baseline deliberately.

Final operating rule: every caller retries at most a little and with jitter; every node and dependency runs under a hard concurrency limit with a short bounded queue and an isolated pool; every dependency call sits behind a narrowly scoped, hysteretic fast-fail with cached fallback; excess load is shed by priority at admission; and recovery is always a monitored probe-and-ramp, never an all-at-once reconnect. Hold the in-flight count inside its bounds and the cascade has nothing to feed on.
===RECOMMENDATION_END===
