Hotspot Containment and Removal¶

Emergency-response protocol — instantiates Overshoot-Crash Load Management

Isolates a zone where collapse has already started and drains the concentrated failing stock through a capped destination before its by-products cascade into the wider system.

Hotspot Containment and Removal is the emergency protocol for the case the earlier controls did not catch: somewhere, local collapse has already begun, and the danger now is propagation — the failing zone's by-products (decomposition, defaults, retries, contamination, panic) spilling into and toppling its neighbours. Its defining move is isolate first, then remove: seal the hotspot so its collapse cannot spread, then draw the concentrated failing stock out through a destination that can actually absorb it. Unlike the planned, value-preserving drawdown of staged harvesting, this is triggered by collapse in progress and optimizes for stopping the spread, not for salvage.

Example¶

A cloud service tips into congestion collapse in one availability zone: a burst of failures triggers client retries, the retries pile into a growing stock of in-flight requests that the zone can no longer drain, and the saturation begins spilling into shared databases every other zone depends on. The on-call protocol is containment-and-removal — trip the circuit breaker to the failing zone (isolation) and shed its accumulated retry load into a rate-limited overflow queue whose capacity is explicitly capped, so the drain does not simply move the collapse into the queue's downstream.

Once the zone stops accepting new load and the backlog is drained within the overflow's cap, the hotspot is contained and the neighbouring zones stay up. The protocol makes no attempt to preserve the failed requests; its whole job is to stop one zone's collapse from becoming everyone's, then hand off to slower recovery once the boundary is holding. The bulkhead between zones is what made the isolation possible in the first place.

How it works¶

Isolate the hotspot first. A boundary — circuit breaker, cordon, quarantine, cohort freeze — goes up around the failing zone before removal starts, so containment does not depend on finishing the cleanup.
Remove the concentration. The dangerous accumulated stock is drawn out or processed where it is densest, because that is where by-products propagate fastest.
Cap the destination. Removed load is routed only as fast as a receiving sink can absorb it without itself overloading; a nominal exit that merely displaces the collapse does not count.
Define the exit. Explicit criteria — authority, handling standards, end conditions — say when the hotspot is contained and the boundary can lift.

Tuning parameters¶

Isolation aggressiveness — how hard and wide the boundary is drawn; tighter contains more surely but strands more that was still healthy inside the cordon.
Removal rate — how fast concentrated stock is drained; faster ends the local hazard sooner but risks overloading the destination sink.
Destination cap — the ceiling on load routed into any one sink, trading containment speed against protecting the downstream system.
Trigger sensitivity — how early "local collapse" is declared; jumpy triggers cordon healthy zones, sluggish ones let the spread escape.
Lift criteria — how conservative the conditions for ending containment are, and whether the zone reopens fully or provisionally.

When it helps, and when it misleads¶

Its strength is that it is the backstop that keeps a local failure from becoming a systemic one — the mechanism that buys the rest of the system time once prevention has already failed. Isolating before removing means the spread stops even when the cleanup is slow.

Its failure modes come from acting under pressure. Over-aggressive cordons strand or kill healthy stock trapped inside the boundary, and removing too fast merely relocates the collapse into whatever sink receives it — the failure the destination cap exists to prevent.^[1] The classic misuse is reaching for containment as a routine tool, normalizing hotspots that better upstream control should have stopped, until the emergency quietly becomes the operating model. The discipline is to cap every destination, size the cordon to the actual hotspot, and treat each activation as evidence that a precautionary control upstream failed.

How it implements the components¶

containment_or_removal_plan — its core: the authority, isolation boundary, handling standards, and end criteria for sealing off a hotspot and removing failing stock.
sink_capacity_threshold — it enforces a cap on where removed stock is routed, so emergency drainage protects the receiving system rather than overloading it.

It does not monitor the residual load that lingers after the hotspot is cleared (Post-Crash Residual-Load Dashboard), perform the planned pre-collapse drawdown (Staged Harvesting or Decommissioning), or verify sink capacity system-wide and ahead of time (Sink Capacity Audit) — it applies a local cap in the moment. It acts only where collapse has already started.

Instantiates: Overshoot-Crash Load Management — the last-resort backstop when growth control and orderly drawdown have already been overtaken.
Consumes: the signal that locates the hotspot (Post-Crash Residual-Load Dashboard, Early Warning Indicator); a legitimate cleared destination to drain into (Clearance Pathway Enhancement).
Sibling mechanisms: Staged Harvesting or Decommissioning · Sink Capacity Audit · Clearance Pathway Enhancement · Cohort Staggering · Controlled Drawdown Schedule · Post-Crash Residual-Load Dashboard · Threshold-Triggered Input Cap · Source Reduction Program · Saturation Dashboard · Early Warning Indicator · Growth-and-Crash Stock-Flow Model · Reentry Gate Review · Secondary-Capacity Reserve Activation

Notes¶

A recurring need for hotspot containment is itself a diagnosis, not just a fix: if local collapses keep appearing, the precautionary band and Threshold-Triggered Input Cap upstream are set too loose. Treat every activation as a signal to tighten the earlier controls, or the protocol slides from backstop to routine.

References¶

[1] Congestion collapse in packet networks — where rising load triggers retransmissions that consume the very capacity needed to recover, driving throughput toward zero — is the canonical case for isolate-and-shed responses such as circuit breakers, load shedding, and bulkheads. It also shows the trap this mechanism must avoid: draining a saturated node into an uncapped neighbour simply moves the collapse rather than ending it. ↩