Runbook Rehearsal¶
Procedure rehearsal — instantiates Chaos Exposure Testing
Executes a documented recovery procedure step by step against a stand-in scenario to find where the written runbook is wrong — missing permissions, ambiguous steps, impossible timing — and drives the corrections back into the document.
A runbook rehearsal puts a written procedure on trial. Someone opens the documented recovery steps — the failover runbook, the restoration procedure, the incident playbook — and actually walks them, in order, against a stand-in scenario, checking whether each step can really be performed as written. Its defining move is that the object under test is the document, not the system and not the reflex: the question is whether the runbook, followed literally by a real person, actually works, or whether it silently assumes an access that expired, a name that changed, a step whose true duration nobody measured. The prize is the defect in the instructions — the gap between what the procedure says and what a person under pressure can do — caught while it is cheap to fix. Where a Fire Drill rehearses a reflex until it is automatic, a runbook rehearsal reads a procedure until it is correct, then edits it.
Example¶
A regional grid operator maintains a black-start runbook — the procedure for restarting the power system from total collapse, when there is no outside electricity to lean on and generators must be brought up in a precise sequence. It has not been executed in years; everyone assumes it works because it exists. In a rehearsal, two senior operators sit with the actual document and a stand-in scenario ("the grid is dark; begin") and walk every step exactly as written. Three defects surface fast. Step 4 requires switching authority that, since a reorganization, now belongs to a control center that closes at night — and a black start at 2 a.m. would stall there. Step 9 says "energize the cranking path" without naming which of two rebuilt substations is now the correct one. And the whole sequence assumes a diesel start time that field crews say is optimistic by twenty minutes. None of these would have appeared in a real black start except as catastrophe. Each becomes a logged correction: the runbook is edited, the missing authority is delegated in advance, the ambiguous step is disambiguated, and the timing is re-baselined — with a note to rehearse again after the fixes land.[n1]
How it works¶
- Follow the document literally. Execute the runbook exactly as written, not as the author remembers meaning it, so the gaps between intent and instruction surface.
- Test the seams the prose hides. Probe the things a document quietly assumes — that the reader still has the access, that the named system still exists, that the stated timing is real.
- Log every defect as it appears. Each ambiguity, missing permission, or impossible step becomes an item, captured in the moment rather than reconstructed later.
- Close the loop into the document. The rehearsal is not done when the walkthrough ends; it is done when the corrections are edited back into the runbook and a re-rehearsal is scheduled.
Tuning parameters¶
- Literalness — how strictly the rehearser follows the exact words versus filling gaps from experience; strict literalness finds more defects but can feel pedantic, lenient reading misses the ambiguity a stressed stranger would hit.
- Rehearser familiarity — run by the author versus by someone who has never seen the runbook; a fresh reader exposes assumed knowledge, an expert reads faster but forgives more.
- Scenario realism — a bare "assume failure X" versus a detailed simulated situation; richer scenarios surface timing and sequencing defects a bare prompt skips.
- Coverage depth — rehearse the whole runbook or only its risky branches; full coverage is thorough but slow, targeted coverage tests the steps most likely to be wrong.
- Correction discipline — whether defects must be edited back before the rehearsal counts as complete; strict close-out prevents the same gap being rediscovered, loose close-out lets findings rot.
When it helps, and when it misleads¶
Its strength is that it validates the artifact everyone silently trusts during a crisis — the written procedure — and it does so cheaply, catching the expired access, the renamed system, and the fantasy timing that turn a documented recovery into a stalled one. Because its whole output is corrections fed back into the document, it directly attacks the archetype's "the runbook exists but nobody knows if it works" failure.
Its central weakness is that it tests the document, not live execution: a runbook can walk perfectly on paper and still fail on contact with the real system, because reading a step is not the same as performing it under load. It is also easily hollowed into a compliance ritual — a walkthrough run to date-stamp the runbook as "reviewed," where defects are noted and never edited back, so the same gap is rediscovered every year. The classic misuse is the rehearsal whose findings die in the meeting notes. The discipline that guards against this is to require every defect be corrected into the document before the rehearsal is closed, to periodically pair the paper walk with a live drill, and to run it with someone who does not already know the answers.
How it implements the components¶
learning_loop— it closes the loop by editing discovered defects back into the runbook and scheduling a re-rehearsal, which is what turns a walkthrough into durable improvement.repair_backlog— each ambiguity, missing permission, and impossible-timing defect is captured as a logged item that outlives the session and drives the fix.
It reads and repairs a known procedure rather than delivering a fault or probing as an adversary: the scripted trigger and bounded scope (perturbation_plan, blast_radius_limit) are Fire Drill's, and the authorized adversarial engagement (exposure_scope_authorization) is its findings-sharing twin Red-Team Stress Test's, which hunts unknown weaknesses where this validates a documented one.
Related¶
- Instantiates: Chaos Exposure Testing — it is the mechanism that puts a documented recovery procedure on trial and repairs it.
- Sibling mechanisms: Fire Drill · Red-Team Stress Test · Disaster Exercise · Chaos Engineering Experiment · Failure Injection · Observability Dashboard · Tabletop Exercise
Editorial Notes¶
Form Classification¶
Form family: Experiment, Test & Rehearsal
Rationale: Runbook Rehearsal operates as an active test, trial, simulation, drill, or rehearsal that generates evidence through a deliberate attempt or perturbation because it executes a documented recovery procedure step by step against a stand-in scenario to find where the written runbook is wrong — missing permissions, ambiguous steps, impossible timing — and drives the corrections back into the document.
Independent corroboration: The frozen evidence defines Runbook Rehearsal as 'Executes a documented recovery procedure step by step against a stand-in scenario to find where the written runbook is wrong — missing permissions, ambiguous steps, impossible timing — and drives the corrections back into the document', so its operative form is Experiment, Test & Rehearsal.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Disaster Management & Risk Reduction
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Multi-domain
Rationale: Executing a documented recovery procedure against a stand-in incident to expose impossible steps, missing permissions, and ambiguity is a preparedness exercise. FEMA HSEEP formalizes exercise design, conduct, evaluation, and improvement planning; organization and computing contribute the runbook content.
Related originating lineages:
- Computer Science & Software Engineering — computer_science contributes algorithms, data models, release isolation, and executable procedures to the mechanism's formative or independently convergent form; that contribution does not displace the primary disaster_management lineage.
- Engineering & Design — Maintenance and commissioning practice materially tests procedural feasibility.
- Organizational & Management Science — Runbook Rehearsal's terminology and operating form—executes a documented recovery procedure step by step against a stand-in scenario to find where the written runbook is wrong — missing permissions, ambiguous steps, impossible…—are rooted most directly in organizational design, management, and operational governance.
- Security Studies & Intelligence Analysis — security_intelligence contributes threat assessment, adversarial probing, escalation, and bounded response to the mechanism's formative or independently convergent form; that contribution does not displace the primary disaster_management lineage.
- Systems Thinking & Cybernetics — Systems thinking, feedback control, and cybernetics supplies a parallel or contributing lineage for the mechanism's defining operation: executes a documented recovery procedure step by step against a stand-in scenario to find where the written runbook is wrong — missing permissions, ambiguous steps, impossible….
Review resolution: The blind reviewers disagreed on primary lineage (disaster_management versus organizational_management); authoritative or primary research supports disaster_management as the best historical origin. Executing a documented recovery procedure against a stand-in incident to expose impossible steps, missing permissions, and ambiguity is a preparedness exercise. FEMA HSEEP formalizes exercise design, conduct, evaluation, and improvement planning; organization and computing contribute the runbook content. The cited FEMA, Homeland Security Exercise and Evaluation Program Doctrine directly supports the defining operation used in that choice. All independently supported contributing domains are retained without an arbitrary cap, while domain_reach=multi_domain records later applicability separately from provenance.
Encyclopedia synthesis: The exact catalogued form synthesizes established practice rather than reproducing a single standard historical label.
Review outcome: Researched adjudication after independent review; high confidence.
Sources consulted:
Notes¶
[n1] A runbook (or playbook) is a documented, step-by-step operating procedure for a recurring or emergency task — a staple of site-reliability and operations practice. Its value depends entirely on being correct and current, which is exactly what a rehearsal that follows it literally is designed to check. ↩