Degraded-Mode Runbook¶

Contingency operating protocol — instantiates Eventual-Occurrence Containment Design

The pre-written procedure for running on reduced capability — which functions to shed, which to keep alive by hand, and the verified path back to full service.

Once the event has happened and the system has dropped into a safe or degraded state, someone still has to run it there. Degraded-Mode Runbook is the pre-written procedure for exactly that stretch: which functions must be kept alive, which get shed, how to operate the survivors by hand when the normal automation is gone, and how to climb back once the failure is cleared. Its distinguishing job is operating within the reduced state and recovering from it — a human procedure decided in calm, before the incident, so the incident is spent executing rather than improvising. That separates it cleanly from the reflex that enters the safe state automatically (the trip) and from the drill that proves the recovery path works; the runbook is the operating plan those bracket on either side.

Example¶

At an air traffic control centre, the primary radar-and-flight-data automation fails mid-shift. The degraded-mode runbook takes over the moment the automation is declared lost. It tells controllers to fall back to procedural separation — spacing aircraft by time and reported position instead of radar returns — and, because that is far more labor-intensive per aircraft, to throttle the traffic acceptance rate, cutting arrivals to roughly half so the critical function, safe separation, is never compromised. Non-essential services (sequencing optimizations, some coordination tasks) are explicitly shed. Crucially, every step is designed to work without the failed automation: the procedures rely on paper strips, direct voice coordination, and standby frequencies — not on the very system that went down. When the automation is restored, the runbook's re-entry checklist governs the hand-back: verify the system, reconcile the traffic picture, then step the acceptance rate back up. Capacity fell; safety did not.

How it works¶

The runbook's real work happens before any incident: it pre-decides the triage. Someone has already ranked functions into must-keep versus can-shed, written the manual workarounds for the must-keeps, and — the load-bearing constraint — verified that none of those workarounds depend on the thing most likely to have failed. It also owns the return: the criteria and steps to climb back to full service safely, rather than lurching back and re-triggering the fault. What it does not do is size the failure or trip the isolation; it assumes the system is already degraded and answers the question the reflex can't — now what, and for how long, and how do we get back.

Tuning parameters¶

Critical-vs-sheddable line — how much to keep alive versus let go. Preserving more maintains service but raises the manual load and the risk of overrunning the crew; shedding aggressively is safer but costs more capability.
Throughput throttle — how far to cut load in degraded mode. A deep cut protects the critical function with margin; a shallow cut preserves service but can swamp the manual process.
Workaround depth — how much can genuinely be done by hand. Deeper manual fallbacks survive worse failures but demand more skill and rehearsal to stay real.
Re-entry criteria — how much verification before returning to full service. Strict criteria avoid a bounce-back into the fault; loose criteria restore capacity faster but risk re-triggering it.
Staffing assumption — who runs it, and whether they are actually available at 3 a.m. A runbook that presumes absent or unpracticed staff is a plan on paper only.

When it helps, and when it misleads¶

Its strength is converting "we'll figure it out" into a rehearsed sequence that keeps the essential function alive when the normal system is gone — the difference between a controlled degradation and a scramble. Its failure modes are almost all forms of containment theater: a runbook written to satisfy an auditor, filed, and never drilled; one that quietly depends on the failed infrastructure it is supposed to survive; one that has drifted out of step with the real system since it was written; or one whose manual workarounds were never actually attempted and turn out to be fantasy under load. The discipline is to drill it under realistic failure — ideally via Failure-Injection Test and recovery drills — and to check, step by step, that nothing in it needs the very thing that broke.

How it implements the components¶

Degraded-Mode Runbook realizes the operate-and-recover slice of the archetype — the parts a human procedure carries, not the parts a machine reflexively enacts:

critical_function_preservation_plan — it names which functions must survive the degraded state and prescribes the manual, dependency-free means of keeping them running.
recovery_and_recurrence_plan — it owns the verified path back: the steps, checks, and re-entry criteria for restoring full service without re-triggering the fault.

It runs and recovers the degraded state but does not enter it automatically (Automatic Isolation Trip), prove the restore path actually works (Recovery Drill and Restore Test), or analyze recurrence after the fact (Post-Incident Recurrence Review).

Instantiates: Eventual-Occurrence Containment Design — the runbook is the archetype's safe-and-degraded-operation layer, turning "enter a safe state" into a plan for actually living in it.
Consumes: Automatic Isolation Trip drops the system into the safe state the runbook then operates from.
Sibling mechanisms: Automatic Isolation Trip · Failure-Injection Test · Recovery Drill and Restore Test · Post-Incident Recurrence Review · Blast-Radius Test · Sentinel Event Monitoring · Cumulative Risk Horizon Table · Fault Tree with Repeated-Opportunity Branch · Probabilistic Safety Assessment · Repeated-Trial Probability Calculator · Opportunity Exposure Register · Stop-or-Scale-Back Gate

Notes¶

A runbook that depends on failed infrastructure is not a safe state — it is a wish. The single most common way a degraded-mode plan fails in the moment is that it silently assumes a resource the incident has already taken out: the dashboard that's down, the network that's cut, the on-call who's unreachable. Every step should be checked against the question "does this still work if the thing that broke is still broken?"

References¶

Graceful degradation — designing a system to drop to reduced but still-useful function under partial failure, rather than failing all at once — is the design property a degraded-mode runbook operationalizes for humans.