# Working — c24 (control policy trace)

## STATE: problem
- **Statement.** A high-traffic service suffers intermittent total outages. A modest request spike slows a few backend nodes; clients/internal callers auto-retry, multiplying load on the struggling nodes, pushing more nodes over the edge until the whole service saturates. Recovery requires operators to manually shed almost all traffic and bring the system back slowly.
- **Constraints.** Can change client + server behavior, set limits, add capacity, has good monitoring; cannot indefinitely over-provision.
- **Success.** A localized slowdown stays localized AND the system self-recovers to normal within minutes without manual shedding.
- **Failure.** A single hot spot can still cascade to total outage, OR safeguards throttle so aggressively that normal traffic is routinely degraded.

## Verbs run (in order, with why)

**1. match / transport (facet retrieval).** Problem is oblique — it instantiates at least four structures at once (positive-feedback amplification, a multiplying-retry regeneration mechanism, a no-autonomous-return tipping/hysteresis trap, and fault-isolation). Ran `search_by_facets` over six domain-stripped facets + a blended `search_prime`. Surfaced, with high confidence: `circular_causality` (the retry→overload→more-retry reinforcing loop; its own text names "retry storms / cascading failure / circuit-breaker patterns as loop interruption"), `cascade`, `tipping_points`/`hysteresis` (asymmetric up/down thresholds → no self-recovery), and the recovery family containing `circuit_breaker` and `controlled_reentry`. Archetypes: `instability_dampening`, `cycle_breaking`.

**2. salience-rank.** Load-bearing: `circular_causality` (names the loop to cut), `circuit_breaker` (the trip-isolate-reset skeleton), `instability_dampening` (the lever menu: rate-limit, backoff, bulkhead, buffer, staged escalation, fallback), `controlled_reentry` (the autonomous-return half — staged ramp, half-open probe, hysteresis), `cycle_breaking` (the retry is the *regeneration point*). Context: `hysteresis` (threshold-gap rationale), `load_shedding`/`bounded_backlog` (priority shedding + bounded queues).

**3. compose → model.** Nodes: client/caller, retry policy, queue, backend node, dependency edge, monitor, breaker, admission controller. Edges: retry→queue (amplifies), queue→node (loads), node-slow→retry (closes the reinforcing loop), breaker⊣loop (cuts), admission⊣queue (shed). Every operative prime annotates an edge — referential integrity holds.

**4. lift L1 (situation→meta-model).** Skeleton: *a shared bounded resource under a reinforcing loop whose gain rises with stress; the loop has an asymmetric activation/recovery threshold so it cannot self-clear; remedy = cut loop gain at its regeneration point, contain locally, then re-engage on a staged, monitored, hysteretic path.* No domain terms remain; every role has a witness in the model.

**5. lift L2 — coverage check (CHOSE TO RUN: yes).** Posed each pattern's components against the draft and checked instantiation:
- *Circuit_breaker* → trip threshold = INSTANTIATED (per-dependency error/latency budget); isolation scope = INSTANTIATED (per-dependency + per-shard bulkheads); reset policy = INSTANTIATED (half-open probe); **over-lift guard / T5** = degraded-but-useful service should not be amputated → handled by serving cached/fallback rather than hard-open where possible.
- *Cycle_breaking* → regeneration point = INSTANTIATED (the automatic retry); interruption lever = INSTANTIATED (retry budget + backoff+jitter); replacement pathway = INSTANTIATED (fail-fast fallback so callers still get an answer); recurrence monitor = INSTANTIATED (retry-ratio + saturation alarms).
- *Instability_dampening* → gain reduction = INSTANTIATED (token-bucket rate limits, concurrency caps); buffer = INSTANTIATED (bounded queues, shed on overflow); propagation boundary = INSTANTIATED (bulkheads); preserve true-signal = INSTANTIATED (priority classes so health-checks/critical paths bypass shedding) — this is the anti-over-throttle guard for the failure criterion.
- *Controlled_reentry* → staged ramp = INSTANTIATED; telemetry gate = INSTANTIATED; re-suspend on relapse = INSTANTIATED; T4 "re-creates original problem" = guarded by ramping at a rate below measured headroom; **hysteresis** (recover threshold stricter than trip threshold) = INSTANTIATED to stop breaker chatter.
- *Load_shedding/bounded_backlog* → admission control at edge + queue caps = INSTANTIATED.
- Correctly DECLINED: indefinite over-provisioning (constraint-barred); broad market-wide-style global halt (isolation must be fine-grained, not whole-service) — declining it IS the fix for the cascade.
- No component left `missing` → STOP condition met.

**6. evaluate-fit (gate).** Anti-signatures walked: "problem is a one-time shock" — not tripped (recurs). "Already-large disturbance needs recovery not sensitivity-reduction" — partially tripped → that's why a recovery (reentry) arm is included alongside the dampening arm. "Damping suppresses legitimate signal" — guarded by priority classes + health-check exemption. Candidates kept.

**7. reconcile.** Concrete model says "cut the retry loop"; meta-model says "reduce loop gain at the regeneration point AND restore via a hysteretic staged path." Divergence: a pure breaker (concrete) would satisfy containment but not autonomous recovery; the meta-model forces the reentry/hysteresis arm. Both readings retained → recommendation carries both arms. Meta-model not decorative.

STOP — sound recommendation ready.

===RECOMMENDATION_BEGIN===
The outages are a self-feeding loop: a few slow nodes cause timeouts, timeouts trigger automatic retries, retries multiply load on exactly the nodes that are already struggling, and the overload spreads until everything is saturated. Because the level of load that ignites the loop is far higher than the level at which it dies down on its own, the system cannot climb back out without a human throwing away nearly all traffic. The design therefore has two jobs that must be built together: starve the amplifying loop at the point where it regenerates, and give the system a staged, self-driven way back.

Start at the retries, because that is where a small fault becomes a flood. Replace blind immediate retries with capped retries that back off and add randomness: at most one or two retries per request, each after an exponentially growing, jittered delay so callers do not resynchronize into waves. Govern retries with a budget — a caller may spend retries on no more than a small fraction (for example 10%) of its recent requests; once that budget is exhausted it stops retrying and fails fast. A failed-fast call should return a cached or default answer where one exists, so that cutting the retry does not simply convert load into errors. This single change removes most of the multiplier.

Wrap every call to a backend or dependency in an automatic trip. Each caller continuously watches the error rate and latency of each dependency; when either crosses a threshold, the caller stops sending new requests to that dependency and immediately serves fallbacks instead of piling up on timeouts. Keep the trip scope narrow — per dependency and, where possible, per shard or partition — so one hot partition is isolated without removing healthy capacity. Crucially, the threshold to re-engage must be stricter than the threshold that tripped it, and re-engagement must be a single trial request (a probe) that is promoted only if it succeeds: this gap prevents the breaker from oscillating rapidly between open and closed, which is worse than either state.

Protect the servers from the inside too. Give each node a bounded concurrency limit and a bounded queue; when the queue is full, reject new work at the door rather than accepting it and slowing everything already in flight. Shed by priority: health checks, critical user paths, and internal control traffic get reserved capacity and are admitted first; bulk, low-priority, and best-effort traffic are the first to be rejected when the node is near its limit. Partition callers and workloads into separate pools (bulkheads) so a flood in one workload cannot consume the threads, connections, or memory that another depends on. The combination of bounded queues, concurrency caps, and priority admission ensures a node degrades gracefully — staying responsive to important work — instead of collapsing.

These guards must not strangle normal traffic, which is the opposite failure. Size every limit from observed normal peaks plus headroom, exempt health checks and critical paths from shedding, and make every trip, shed, and rejection visible in monitoring with its reason, so a too-tight limit is caught and retuned rather than silently degrading service.

Recovery should be automatic and gradual, never a manual cliff. When a node or dependency recovers, do not return it to full traffic at once. Ramp it: admit a small share of traffic, watch error rate and latency for a short interval, and only increase the share if the metrics hold; if they degrade, step back down automatically. Pair this with autoscaling that adds capacity when sustained healthy load approaches limits — within a ceiling, since hardware cannot grow without bound — so the staged ramp has somewhere to grow into. The ramp rate must stay below the measured spare headroom, or the act of returning traffic will reignite the very loop you just broke.

The final operating rule: never let a failed call generate more total load than the original call, and never return a recovering component to full load faster than it can be proven to absorb. Cut the retry multiplier at its source, isolate faults narrowly with hysteretic trips, shed and queue by priority so nodes degrade instead of dying, and bring everything back on a monitored ramp that backs off at the first sign of relapse. A local slowdown then stays a local slowdown, and the system climbs back on its own within minutes.
===RECOMMENDATION_END===

COST: ~8 operations (2 facet/semantic searches, 4 get_prime/get_archetype, 1 archetype search, 1 facet search), ~32000 tokens.
