Skip to content

Containment or Rollback Action

A reversible stabilizing action — instantiates Acute Stabilization Command

Stops the bleeding by isolating the blast radius or reverting to the last known-good state — a deliberately reversible move that buys time without committing to a cause.

A Containment or Rollback Action is the concrete move that stops the bleeding without pretending to cure the patient: draw an isolation boundary around the failing part so the damage can't spread, or revert to the last state known to be good. Its defining property is deliberate reversibility — it is chosen precisely because it can be undone, which is what makes it safe to do before the cause is understood. It buys time and shrinks the blast radius; it does not fix root cause and does not claim to. This is the archetype's "act before full diagnosis" license made physical, and its whole discipline is refusing the tempting irreversible "real fix" while the situation is still unstable.

Example

A new software release goes out at noon; within minutes, checkout-error rates climb. The responder does not begin debugging the new code in production. She takes a containment/rollback action: the feature is disabled behind its flag (an isolation boundary — the bad path is fenced off) and the deployment is rolled back to the previous known-good build. Error rates fall within minutes. Nobody yet knows why the release broke checkout — and that is the point. The move was chosen because it is reversible and shrinks the blast radius, so it can be taken immediately, on incomplete information, without betting the system on a guess.

Contrast the move it refuses: hot-patching the new code live to "just fix it." That might work, but it is irreversible under pressure and commits to an unverified cause. Containment buys the time to diagnose calmly later; the rollback is the safe default precisely because it can be undone.

How it works

  • Isolate or revert, always reversibly. Either fence the failing component behind a containment boundary (kill switch, quarantine, cordon) or return to the last known-good state — chosen from a set of pre-approved, undoable moves.
  • Shrink blast radius first. The goal is to stop spread and stabilize, buying time; the correctness of the underlying system is explicitly deferred.
  • Preserve the ability to diagnose. Because the action is reversible and non-destructive, it doesn't erase the evidence a later root-cause analysis will need.
  • Prefer the smallest sufficient move. Contain narrowly if that holds; escalate to a full rollback only when isolation isn't enough.

Tuning parameters

  • Blast-radius aggressiveness — how much you isolate or roll back. A wide rollback restores stability fast but reverts good changes too; a narrow cordon preserves more but may not fully stop the spread.
  • Reversibility bar — how strictly a move must be undoable before it's allowed pre-diagnosis. A high bar keeps options open (safest) but may exclude a faster fix; a lower bar acts harder but risks a one-way door.
  • Trigger autonomy — whether responders may contain on their own authority or must get command sign-off. Autonomy is faster; sign-off is safer for high-impact reversions.
  • Evidence preservation — how much you snapshot before reverting, trading a few seconds now for a diagnosable trail later.

When it helps, and when it misleads

Its strength is that it lets a team act now, on incomplete information, without gambling — because a reversible move can be wrong and still recovered from. Isolating the blast radius and reverting to known-good is often the single fastest route back to stability, and it keeps the harder question (what actually broke) for a calmer moment.

Its failure modes turn on the reversibility claim being false. A move labelled reversible that isn't — a "rollback" that drops data, a cordon that silently corrupts state — is worse than doing nothing, because it was taken on the assumption it could be undone. The classic misuse is reaching past containment for the irreversible "real fix" under pressure, committing an unstable system to an unverified cause. And repeated rollback can mask a worsening root problem, treating the symptom until the underlying fault grows. The discipline that guards against this is verifying reversibility before acting, preserving evidence as you go, and treating every containment as buying time for diagnosis — never as the diagnosis itself.[1]

How it implements the components

  • containment_and_isolation_boundary — draws the boundary that fences the failing part off from the rest, stopping the spread.
  • reversible_degradation_option_set — selects and executes one move from the menu of pre-approved, undoable actions (kill switch, revert, cordon).

It executes reversible moves but does not author the menu or the floor they respect — those are written in the Incident Response Runbook; graceful load-shedding toward the floor (rather than isolate/revert) is its sibling Reversible Service Degradation; and the authority for a high-impact action comes from Incident Command System.

  • Instantiates: Acute Stabilization Command — the reversible stop-the-bleeding move at the centre of stabilization.
  • Consumes: Incident Response Runbook supplies the sanctioned option set and the floor; Incident Command System authorizes high-impact reversions.
  • Sibling mechanisms: Reversible Service Degradation · Incident Response Runbook · Incident Command System · Common Operating Picture Board · Incident Action Log · On-Call Rotation Activation · Severity Matrix Activation · Triage and Prioritization Protocol · Status Update Cadence · War Room or Incident Channel · Root Cause Analysis Handoff · Deactivation Checklist · Post-Incident Review Hotwash

Notes

Easily conflated with Reversible Service Degradation: both are reversible and both buy time, but they answer different questions — containment/rollback asks "how do I stop this from spreading, or undo it?" while degradation asks "how do I keep serving within the floor while impaired?" A response often uses both at once.

References

[1] "Blast radius" — the scope of damage a failure, or a corrective action, can cause — is a standard reliability-engineering framing; containment works by deliberately shrinking it, and a good rollback is designed to be reversible precisely so its own blast radius stays near zero.