# Working — control-policy trace

## problem
Intermittent total outages from a metastable congestion collapse: a modest load spike slows a few backend nodes; clients + internal callers auto-retry failed requests; retries re-emit load onto the already-slow nodes; more nodes tip; the whole service saturates. It does NOT self-recover — only manual shedding clears it. Levers: change client/server behavior, set limits, add capacity (but not over-provision indefinitely). Success = a local slowdown stays local AND the system returns to normal without manual shedding within minutes; failure = a hot spot can still cascade, OR safeguards throttle so hard that normal traffic is routinely degraded.

## verbs run

**1. match (retrieve).** corpus_stats (sanity: 643 primes, semantic engine live). search_prime with the domain-stripped meta-model ("local perturbation → self-amplifying loop where the corrective response adds load → tipping point → saturated collapsed state that won't self-recover"). Top structural hits: `cascade`, `tipping_points_or_phase_transitions`/`regime_change`, `hysteresis`, `circular_causality`/`feedback`, `homeostasis`/`ultra_stability`.

**2. match via search_by_facets** (problem is oblique — instantiates ≥5 structures at once). Per-facet reads surfaced what a blended query buries:
- "corrective response amplifies the stress it relieves" → the retry-as-positive-feedback core.
- "self-reinforcing loop past a tipping point" → `tipping_points`, `regime_change`, `circular_causality`.
- "two stable states, trapped in the bad one once a threshold crosses" → `threshold_and_criticality`, **hysteresis** (load-bearing: explains why no self-recovery).
- "local failure propagates through coupling until saturated" → `cascade`, `systemic_risk`, `coupling`.
- "no limit on demand admitted to a contended shared resource" → `margin_of_safety`, `load_balancing` (the missing admission control).
- "cannot return to normal on its own; needs external intervention" → `resilience`, `controlled_reentry`, `irreversibility`.
- "isolate a failing part so failure doesn't spread" → `systemic_risk`, isolation/firebreak.

**3. match (archetypes).** search_archetype on the meta-model → `instability_dampening` (0.76), `tipping_point_prevention` (0.76), `balancing_loop_stabilization` (0.76), `saturation_avoidance`, `fail_safe_default`, `coupling_calibration`, `controlled_reentry`. Pulled full records for cascade (prime), hysteresis (prime), instability_dampening, balancing_loop_stabilization, tipping_point_prevention.

**4. salience-rank / prune.** Load-bearing: `cascade` (the re-emission mechanism — retries are the re-emitter), `circular_causality`/`feedback` (the loop polarity), `hysteresis` (why recovery needs to undershoot), `instability_dampening` (prevention pattern), `balancing_loop_stabilization` (autonomous-recovery pattern). Context only, declined to add: `systemic_risk` (single-service, not inter-institution), `controlled_reentry` (a recovery flavor — its useful part folds into the recovery loop), `saturation_avoidance` (receptor-saturation framing subsumed by admission control). Pruned to the two operative archetypes; cascade/hysteresis are the diagnostic primes that tell us WHICH components are load-bearing.

**5. compose (referential integrity).** Model nodes: {client/caller, edge-limiter, backend-node, shared-resource, health-signal, controller}. Edges: client →(emits)→ node; node →(on-slow, fails)→ client; client →(retry = RE-EMIT, annotated `cascade`/`feedback`)→ node (the +loop); limiter →(caps admitted)→ node (annotated `instability_dampening`: propagation boundary + gain); controller →(senses health, sheds, re-arms)→ limiter (annotated `balancing_loop_stabilization`); node →(metastable bad state)→ node (annotated `hysteresis`). Every operative prime appears as an annotation; every edge connects declared nodes. ✓

**6. lift L1 (situation → skeleton).** Roles: SOURCE (re-emits on failure), CHANNEL (limit-gated coupling), CONTENDED-RESOURCE (thresholded, tips super-critical), SENSOR, CONTROLLER (closes a balancing loop). Invariants: keep multiplication-factor < 1 (sub-critical); keep admitted demand ≤ serviceable; recovery line set BELOW collapse line. No domain words; each role has a witness in the model. ✓

**7. lift L2 — COVERAGE MAP (below).**

**8. evaluate-fit (anti-signature gate).**
- instability_dampening anti-sigs: "problem is slow beneficial spread" — NO. "pure absence of capacity" — NO (it's response dynamics, and over-provisioning is explicitly off the table). "stable but inefficient; need redistribution" — NO. "disturbance already large, need recovery not sensitivity-reduction" — partially true at peak, which is why we ALSO carry the recovery loop. "target should amplify for mobilization" — NO. Not disqualified.
- balancing_loop_stabilization anti-sigs: "only map, don't change" — NO. "no observable deviation signal" — NO (good monitoring exists). "no actuator" — NO (can change client/server behavior, set limits). "contested/adaptive target" — NO (normal-service is a clear setpoint). Not disqualified. Over-damping is the named failure mode → directly addressed by deadband + load-proportional shedding so normal traffic isn't routinely degraded.

## COVERAGE MAP

Operative pattern A — **Instability Dampening** (prevention: keep a local slowdown sub-critical):
- Perturbation Input Definition → **instantiated**: the perturbation class = a few nodes crossing a latency/error threshold under a modest spike.
- Amplification Pathway Map → **instantiated**: identified loop = failed request → automatic retry → extra load on the slow node → more failures. Retries are the re-emitter (the cascade's defining feature).
- Sensitivity/Gain Assessment → **instantiated**: gain = retries-per-failure × fan-out of internal callers; load multiplies geometrically once nodes slow.
- Gain Adjustment → **instantiated**: per-client retry budget (retries as a fraction of successful requests, e.g. ≤10%); exponential backoff with jitter; stop retrying non-idempotent/likely-doomed calls.
- Damping Rule → **instantiated**: shed/limit load proportional to measured pressure, not all-or-nothing.
- Propagation Boundary → **instantiated**: per-dependency bulkheads (isolated connection/concurrency pools) so one hot dependency can't drain shared capacity; circuit-breaker opens an unhealthy downstream path.
- Buffer/Absorptive Capacity → **instantiated**: bounded queues with concurrency limits; small headroom + fast autoscale rather than indefinite over-provision.
- Response Cadence Control → **instantiated**: backoff + debounce on breaker state; admission decisions on a fast control interval.
- Safe Fallback State → **instantiated**: load-shedding returns a fast cheap rejection (e.g. retry-after) instead of a slow timeout that ties up resources; serve degraded/cached responses where possible.
- Residual Instability Monitor → **instantiated**: track goodput, shed rate, breaker state, retry ratio; alarm on both runaway retries AND over-shedding.

Operative pattern B — **Balancing Loop Stabilization** (autonomous recovery without manual shedding):
- Target State/Viable Range → **instantiated**: keep node utilization/latency inside a healthy band (the setpoint = normal service).
- Deviation Signal → **instantiated**: real-time latency/error/queue-depth/concurrency at each node and the admission point.
- Feedback Return Path → **instantiated**: health signal feeds the local admission controller and the breaker — no human in the loop.
- Correction Rule → **instantiated**: adaptive concurrency limit (AIMD-style: additive increase when healthy, multiplicative decrease on overload).
- Corrective Actuator → **instantiated**: the admission limiter / breaker / load-shed gate that actually moves admitted load.
- Loop Gain/Response Strength → **instantiated**: tuned decrease factor on overload, conservative additive recovery.
- Delay Profile → **instantiated**: short sense-act interval; probe-based half-open recovery to avoid acting on stale state.
- Deadband/Tolerance Band + **Hysteresis Band** → **instantiated** (this is the load-bearing reconcile move): separate trip and re-arm thresholds — recover only after pressure falls BELOW a lower line, so the loop doesn't chatter and, crucially, breaks the metastable trap.
- Stability Boundary / Saturation Limit → **instantiated**: hard admission ceiling = the breaker's last resort when adaptive control saturates.
- Effect Monitoring → **instantiated**: confirm shedding raises goodput and that re-admission doesn't re-trip; auto-revert if it does.
- Manual Override/Escalation Path → **correctly_declined as a primary mechanism**: the success criterion is autonomous recovery, so the operator override is retained only as a backstop, not as the recovery path — making it primary would reproduce the failure being fixed.
- Disturbance Model → **instantiated**: tuned against the modest-spike + retry-amplification scenario.
- Anti-Gaming Safeguard → **correctly_declined**: cooperative internal/client population, not strategic actors gaming a metric; forcing it adds machinery the problem doesn't need.

Diagnostic prime **hysteresis** → **instantiated via the trip/re-arm split**: the collapse threshold ≠ recovery threshold is exactly why manual under-shedding was previously required; the controller now supplies that undershoot automatically.

No component left **missing**. Compose integrity ✓, L1 witnesses ✓, L2 coverage complete ✓, gate passed ✓ → STOP permitted.

COST: ~7 operations (6 encyclopedia calls + 1 stats), ~17k tokens.

===RECOMMENDATION_BEGIN===
The outage is not caused by the size of any spike; it is caused by the system feeding its own distress. When a few nodes slow, every failed call provokes automatic retries, and those retries land on the very nodes that are already struggling — so the response to overload becomes a fresh source of overload. Past a certain point the whole service flips into a saturated state that will not clear itself, because the load needed to keep it jammed is far lower than the load that first jammed it. The fix has two halves: stop the amplification before it tips, and give the system a way to climb back out on its own.

First, cut the amplification. Give every client and internal caller a retry budget — retries may consume only a small fraction of its successful-request volume (start near ten percent) — and have it stop retrying once that budget is spent. Use exponential backoff with randomized jitter so retries spread out instead of arriving in synchronized waves, and never retry requests that aren't safe to repeat or are already almost certain to fail. This alone keeps a slow node from being hammered by the traffic meant to recover from its slowness.

Second, put a guard in front of each backend that limits how much work it admits, and let that limit move with the node's own health. Each node continuously reports its latency, error rate, queue depth, and in-flight concurrency. An adaptive admission controller raises the allowed concurrency gently while the node is healthy and cuts it sharply the moment latency or errors climb — additive increase, multiplicative decrease — so admitted demand tracks what the node can actually serve. When demand still exceeds capacity, the guard sheds the excess by rejecting requests immediately with a clear "retry after" signal rather than letting them queue into slow timeouts that pin resources. A fast, cheap rejection protects far more than a slow success.

Third, contain blast radius. Give each downstream dependency its own isolated pool of connections and concurrency slots, so one hot dependency cannot starve the capacity that healthy paths need. Wrap each dependency in a breaker that opens when its error or latency crosses a limit, fails fast while open, and probes for health with a trickle of test traffic before reopening. A local problem is then walled off instead of draining the whole service.

The recovery half hinges on one deliberate asymmetry. Use two different thresholds: trip protection at one level, but only relax it after pressure has fallen well below a second, lower level. Because the jammed state is self-sustaining, simply returning load to the level that the node could "normally" handle will re-trip it instantly — which is exactly why manual recovery historically required shedding almost everything. By making the controller re-admit traffic only after the node has genuinely recovered headroom, and then ramp admission back up gradually through the breaker's half-open probe rather than all at once, the system walks itself out of the bad state in the same way an operator would, but in seconds and without a human. The gap between the two thresholds, plus a small dead-band, also stops the guard from flapping open and shut around the boundary.

Keep capacity elastic but bounded: hold a little headroom and autoscale quickly on sustained healthy demand, rather than trying to over-provision against the worst case. The point of the controls is that you no longer need to.

Finally, watch the controls themselves. Track goodput (useful work completed), the shed rate, retry ratios, and breaker state. Two failure directions matter equally: runaway retries or rising shed-with-low-goodput means the damping is too weak; routine rejection of healthy traffic means it is too strong. Tune the retry budget, the decrease factor, and the trip/re-arm gap against this signal, and keep an operator override only as a last-resort backstop, never as the normal recovery path.

Operating rule: admit only what the slowest healthy node can actually serve, never retry faster than you succeed, and only re-open the gates once pressure has dropped below the line — not merely back to it.
===RECOMMENDATION_END===
