Site Reliability Engineering¶
Beyer, B., Jones, Chris, Petoff, Jennifer, & Murphy, N. R. (2016). Site Reliability Engineering: How Google Runs Production Systems. O'Reilly.
Cited by¶
80 citations across 80 artifacts.
Each citation links to the sentence it supports in the citing article.
Primes¶
- Absence as Information
- The Inference treats the gap as primary signal: during an incident, "the database heartbeat stopped at 02:14" is often more diagnostic than any signal that did arrive, pinning the failure in time and component.
This sourceEstablishes monitoring practice in which a missing expected signal (e.g., a stopped heartbeat) is often the most diagnostic indicator during an incident.
- The Inference treats the gap as primary signal: during an incident, "the database heartbeat stopped at 02:14" is often more diagnostic than any signal that did arrive, pinning the failure in time and component.
- Anna Karenina Principle
- and in software deployment a release succeeds only when build, tests, configuration, dependencies, infrastructure, and rollout all pass.
This sourceThe release-engineering and testing chapters present a production release as gated on a conjunction of necessary conditions — build, automated tests ("push on green"), configuration, and staged/canary rollout — any one of which can block or roll back the deployment.
- and in software deployment a release succeeds only when build, tests, configuration, dependencies, infrastructure, and rollout all pass.
- Baseline Deviation
- In software reliability engineering it is invariant violations, latency excursions, and error-rate anomalies surfaced by observability platforms.
This sourceDescribes monitoring, alerting on latency and error-rate anomalies against baselines, and the problem of baselines drifting to absorb degradation.
- In software reliability engineering it is invariant violations, latency excursions, and error-rate anomalies surfaced by observability platforms.
- Boundedness
- Reliability engineering and site-reliability engineering (SRE) use boundedness as the central architectural-design principle for fault-tolerant systems, as Beyer, Jones, Petoff, and Murphy (2016) document in the canonical exposition of the SRE framework.
This sourcecanonical SRE exposition with explicit boundedness commitments — bounded blast radius, bounded latency, bounded resource use, error budgets (bounded failure rate), recovery-time objectives.
- Reliability engineering and site-reliability engineering (SRE) use boundedness as the central architectural-design principle for fault-tolerant systems, as Beyer, Jones, Petoff, and Murphy (2016) document in the canonical exposition of the SRE framework.
- Callback
- Operations and incident response — standing orders, runbooks, dead-man switches, and escalation policies, all pre-committed handlers held by an operational structure.
This sourceStandard reference treating runbooks, on-call escalation policies, and standing operational procedures as pre-committed handlers invoked by an operational structure when conditions arise.
- Operations and incident response — standing orders, runbooks, dead-man switches, and escalation policies, all pre-committed handlers held by an operational structure.
- Commitment
- And the manufacturing discipline of formal design-freeze and sign-off transferred into release engineering and into error-budget and release-gating practice.
This sourceDescribes error budgets and release-gating as commitment points that convert tentative changes into operative, relied-upon ones.
- And the manufacturing discipline of formal design-freeze and sign-off transferred into release engineering and into error-budget and release-gating practice.
- Configuration Drift
- The asymmetry of update cost is the engine — the operational change took ten seconds and worked immediately, while updating the record is a separate pull-request with review, easily skipped under pressure.
This sourceDocuments change management and the review overhead of reconciled changes versus the speed of an ad-hoc operational fix (toil, configuration-as-code, progressive rollout) — the asymmetric cost that drives drift.
- The asymmetry of update cost is the engine — the operational change took ten seconds and worked immediately, while updating the record is a separate pull-request with review, easily skipped under pressure.
- Controlled Reentry
- Reentry is the moment where maximum information is needed (for decision-making) but minimum margin exists (for error)—a property Beyer et al. (2016) emphasize in production-system rollout design.
This sourceCodifies error budgets, telemetry, and rollback gating for production rollouts/recoveries.
- Reentry is the moment where maximum information is needed (for decision-making) but minimum margin exists (for error)—a property Beyer et al. (2016) emphasize in production-system rollout design.
- Diagnostically Inert Signal
- The minimal repair content is identifiable: what failed (target identification), why it failed (cause attribution), what to do next (action specification), and the escalation path (when this exceeds the recipient's role).
This sourceThe monitoring/alerting chapters argue that an effective alert must be actionable — carrying what failed and what to do — not merely announce a condition; the SRE basis for the minimal repair-content requirement of a failure signal.
- The minimal repair content is identifiable: what failed (target identification), why it failed (cause attribution), what to do next (action specification), and the escalation path (when this exceeds the recipient's role).
- Efference Copy
- A structurally identical applied instance is software deploy management, where a release emits an expected-effect manifest in parallel with the change, so the observability system attenuates the predicted metric movement and alerts only on deviations — the deploy notification playing the role of the command copy.
This sourceTreats change/deploy as the dominant source of production incidents and documents annotating deploys so monitoring attributes consequent metric movement to the self-caused change rather than alerting on it.
- A structurally identical applied instance is software deploy management, where a release emits an expected-effect manifest in parallel with the change, so the observability system attenuates the predicted metric movement and alerts only on deviations — the deploy notification playing the role of the command copy.
- Event Lifecycle Phases
- Software reliability: reliability engineering / on-call response / post-mortem and follow-up, with the blameless post-mortem existing because the post-event phase designs the next pre-event phase.
This sourceDefines reliability engineering, on-call incident response, and the blameless postmortem whose purpose is to improve the next pre-incident phase.
Supported in partVerified against the work's full text
The SRE chapter covers on-call intervention, blameless postmortems and follow-up actions to prevent recurrence, but not the claimed pre-event/post-event lifecycle rationale.
“Blameless postmortems are a tenet of SRE culture.”
- Software reliability: reliability engineering / on-call response / post-mortem and follow-up, with the blameless post-mortem existing because the post-event phase designs the next pre-event phase.
- Habit
- In software and infrastructure, scheduled jobs, default configurations, and automated playbooks are habits in the technical sense, with the devaluation-robustness failure mode appearing as the production incident where automation keeps firing after its purpose has changed.
This sourceChapters on monitoring, practical alerting, and being on-call cover threshold-triggered paging of on-call engineers and remediation (deleting, retuning, downgrading) of stale alerts.
- In software and infrastructure, scheduled jobs, default configurations, and automated playbooks are habits in the technical sense, with the devaluation-robustness failure mode appearing as the production incident where automation keeps firing after its purpose has changed.
- Improvisation
- Software incident response. On-call engineers handle novel incidents with runbooks (the backbone) and prior-incident repertoires (the vocabulary), generating hypotheses and tests against the system's responses.
This sourceOn-call incident handling via runbooks plus prior-incident repertoire, generating and testing hypotheses against the system's responses; blameless postmortems and game-days.
- Software incident response. On-call engineers handle novel incidents with runbooks (the backbone) and prior-incident repertoires (the vocabulary), generating hypotheses and tests against the system's responses.
- Incident Response
- Site reliability engineering: mitigate before resolve; MTTA and MTTR are the acute-phase metrics, and root-cause analysis lives in the post-mortem after the page is closed.
This sourceMitigate-before-resolve, MTTA/MTTR as acute-phase metrics, blameless post-mortem for root-cause analysis, and an incident-command model explicitly modeled on the fire service's ICS.
Supported in partVerified against the work's full text
The SRE postmortem chapter confirms root causes are handled in the postmortem and that mitigation is a distinct action, but says nothing of MTTA/MTTR or an explicit mitigate-before-resolve rule.
“A postmortem is a written record of an incident, its impact, the actions taken to mitigate or resolve it, the root cause(s), and the follow-up actions to prevent the incident from recurring.”
- Site reliability engineering: mitigate before resolve; MTTA and MTTR are the acute-phase metrics, and root-cause analysis lives in the post-mortem after the page is closed.
- Inverted Pyramid
- A 3 a.m. pager alert is built as an inverted pyramid by necessity: the content set is the incident's facts, the importance ordering is severity-and-action, so the alert leads with what is broken, how bad, and what to do — "Payments API down, P1, run the failover runbook" — and only then descends into diagnostic detail, logs, and dashboards.
This sourceEstablishes on-call alerting and incident-response practice in which a page leads with severity, scope, and the action required (the actionable prefix) before diagnostic detail.
- A 3 a.m. pager alert is built as an inverted pyramid by necessity: the content set is the incident's facts, the importance ordering is severity-and-action, so the alert leads with what is broken, how bad, and what to do — "Payments API down, P1, run the failover runbook" — and only then descends into diagnostic detail, logs, and dashboards.
- Linearity
- Listed in the references but not attached to a specific claim.
- Local Autonomy & Tiered Escalation
- As Beyer, Jones, Petoff, and Murphy (2016) document for site reliability engineering, this model pervades modern infrastructure—from incident-response runbooks that route alerts through on-call support tiers, to customer-support systems where Tier-1 agents handle routine requests and escalate to specialists, to microservices architectures where fault isolation and circuit-breaker patterns mirror bureaucratic subsidiarity.
This sourceCanonical SRE text defining the four golden signals (latency, traffic, errors, saturation) and the operational practice of metric collection, alerting, SLO/SLI tracking, and incident response in large-scale software systems.
- As Beyer, Jones, Petoff, and Murphy (2016) document for site reliability engineering, this model pervades modern infrastructure—from incident-response runbooks that route alerts through on-call support tiers, to customer-support systems where Tier-1 agents handle routine requests and escalate to specialists, to microservices architectures where fault isolation and circuit-breaker patterns mirror bureaucratic subsidiarity.
- Loss And Damage
- And in software reliability, error budgets are an explicit residual-harm construct — the outages that will leak past testing and review, budgeted in advance.
This sourceDefines the error budget — the explicitly budgeted residual unavailability (e.g. 0.1% for a 99.9% target) absorbed by the team and used to gate releases when exhausted.
Supported in partVerified against the work's full text
The held chapter says outages are inevitable and can occur despite thorough testing and impact users; it does not define the error budget or the 0.1% for 99.9% arithmetic.
“Incidents and outages are inevitable given our scale and velocity of change.”
- And in software reliability, error budgets are an explicit residual-harm construct — the outages that will leak past testing and review, budgeted in advance.
- Measurement Uncertainty and Observational Noise
- Listed in the references but not attached to a specific claim.
- Memory Consolidation
- The stabilization mechanism is the post-mortem itself: a scheduled replay event that re-presents the timeline, integrates it with prior incidents and existing runbooks, and transfers it from individual memory to a durable institutional store (the runbook, the alerting rule, the regression test).
This sourceFrames the blameless incident postmortem as a scheduled replay event transferring vivid, fragile individual knowledge into durable institutional stores (runbooks, alerting rules, regression tests).
Supported in partVerified against the work's full text
The SRE chapter describes postmortems as written, reviewed records with action items filed in an organization repository of past incidents, but not the replay/runbook/alerting-rule/regression-test framing.
“Once those involved are satisfied with the document and its action items, the postmortem is added to a team or organization repository of past incidents.”
- The stabilization mechanism is the post-mortem itself: a scheduled replay event that re-presents the timeline, integrates it with prior incidents and existing runbooks, and transfers it from individual memory to a durable institutional store (the runbook, the alerting rule, the regression test).
- Monitoring
- The practice integrates signal interpretation, threshold comparison, alerting logic, and the decision to escalate or act, as Beyer et al. (2016) describe in the SRE canon.
This sourceCanonical SRE text defining the four golden signals (latency, traffic, errors, saturation) and the operational practice of metric collection, alerting, SLO/SLI tracking, and incident response in large-scale software systems.
- The practice integrates signal interpretation, threshold comparison, alerting logic, and the decision to escalate or act, as Beyer et al. (2016) describe in the SRE canon.
- Observability
- Listed in the references but not attached to a specific claim.
- Predictive Coding
- Listed in the references but not attached to a specific claim.
- Preparation
- The idle state is cold capacity — servers powered down or unprovisioned, cheap to hold but slow to bring up (minutes to boot, install, warm caches).
This sourceDocuments holding warm/spare capacity to absorb load, where cold capacity incurs multi-minute provisioning/cold-start latency while pre-warmed standby responds near-instantly.
- The idle state is cold capacity — servers powered down or unprovisioned, cheap to hold but slow to bring up (minutes to boot, install, warm caches).
- Prioritization
- Severity tiers are remarkably consistent across companies, suggesting they converge on the right metric—a convergence Beyer, Jones, Petoff, and Murphy (2016) document in the Google SRE incident-management framework that has become a de facto industry reference.
This source(Canonical exposition of the Site Reliability Engineering framework with explicit boundedness commitments throughout: bounded blast radius via cell-based and bulkhead-pattern architecture; bounded latency via timeouts and deadline propagation; bounded resource use via per-tenant quotas; bounded failure rate via SLO/SLI/SLA error budgets; bounded recovery time via recovery-time-objective planning; bounded-throughput rate limiters; bounded-failure-rate circuit breakers; bounded-attempt retry policies. The book is the reference for the operational discipline of reliability engineering as a discipline of boundedness.)
- Severity tiers are remarkably consistent across companies, suggesting they converge on the right metric—a convergence Beyer, Jones, Petoff, and Murphy (2016) document in the Google SRE incident-management framework that has become a de facto industry reference.
- Rate Coding
- The window-set precision-versus-latency trade-off is the operator's live knob: a short window (5 s) reacts fast but is noisy and trips false alarms, a long window (5 min) is precise and stable but slow to detect a spike — the same \(1/\sqrt{N}\) law as the converter.
This sourceStandard reference for monitoring by windowed event-rate aggregation: a short window reacts fast but is noisy, a long window is stable but slow — the precision-versus-latency trade of rate-threshold alerting.
- The window-set precision-versus-latency trade-off is the operator's live knob: a short window (5 s) reacts fast but is noisy and trips false alarms, a long window (5 min) is precise and stable but slow to detect a spike — the same \(1/\sqrt{N}\) law as the converter.
- Refractory Period
- Resource Management
- Overcommit Increases Utilization but Risks Contention** (a tension Beyer, Jones, Petoff, and Murphy (2016) discuss as central to running Google's services at high utilization through statistical multiplexing while preserving safety margin):
This source(Canonical exposition of the Site Reliability Engineering framework with explicit boundedness commitments throughout: bounded blast radius via cell-based and bulkhead-pattern architecture; bounded latency via timeouts and deadline propagation; bounded resource use via per-tenant quotas; bounded failure rate via SLO/SLI/SLA error budgets; bounded recovery time via recovery-time-objective planning; bounded-throughput rate limiters; bounded-failure-rate circuit breakers; bounded-attempt retry policies. The book is the reference for the operational discipline of reliability engineering as a discipline of boundedness.)
- Overcommit Increases Utilization but Risks Contention** (a tension Beyer, Jones, Petoff, and Murphy (2016) discuss as central to running Google's services at high utilization through statistical multiplexing while preserving safety margin):
- Rules Of Engagement
- And software incident management imports it directly — "when a Sev-1 is declared, the on-call may roll back deploys, page the VP, and contact customers without consultation."
This sourceDescribes incident-management roles and pre-authorized actions (e.g., declaring a major incident and rolling back without escalation), the software analogue of rules of engagement.
- And software incident management imports it directly — "when a Sev-1 is declared, the on-call may roll back deploys, page the VP, and contact customers without consultation."
- Single Point of Failure
- In software and distributed systems, a single load balancer with no failover, a master database with no replica, or a key authentication service can take the whole product offline.
This sourceThe canonical SRE reference treating single points of failure (unreplicated load balancers, master datastores, central services) and redundancy (N+1, replication, failover) as the core of production reliability.
- In software and distributed systems, a single load balancer with no failover, a master database with no replica, or a key authentication service can take the whole product offline.
- Swiss Cheese Model (Layered Defense with Aligning Holes)
- In software reliability, unit tests, integration tests, review, static analysis, canary deploys, and monitoring each catch most bugs, and outages occur when a bug threads the holes across all layers.
This sourceStacked testing, canarying, release, and monitoring layers each catch most defects, with outages occurring when a fault threads the holes across all layers.
- In software reliability, unit tests, integration tests, review, static analysis, canary deploys, and monitoring each catch most bugs, and outages occur when a bug threads the holes across all layers.
- Two-Store Architecture
- The transfer mechanism is the post-incident review and documentation sprint, which must run in a dedicated window because, like sleep-dependent consolidation, it competes with normal operation and cannot happen continuously during firefighting.
This sourceFrames the blameless postmortem as a scheduled transfer of incident knowledge into durable institutional memory so failures do not recur.
Supported in partVerified against the work's full text
The SRE chapter supports postmortems as the way incident lessons are captured into a durable repository. It does not say the review needs a dedicated window or competes with firefighting.
“Once those involved are satisfied with the document and its action items, the postmortem is added to a team or organization repository of past incidents.”
- The transfer mechanism is the post-incident review and documentation sprint, which must run in a dedicated window because, like sleep-dependent consolidation, it competes with normal operation and cannot happen continuously during firefighting.
Domain-specific¶
- Fallacy of One Administrator
- SRE inherits it wholesale (service ownership, error budgets, the "who owns this dependency" question are the corrective institutionalized)
Supported in partVerified against the source
- SRE inherits it wholesale (service ownership, error budgets, the "who owns this dependency" question are the corrective institutionalized)
- Service Level
- The portable inferences carry too: the non-linear cost of each additional nine, the √N pooling benefit, the substitutability of inventory, capacity, and redundancy buffers for the same level, and the commitment-device effect by which a published threshold reads differently from an unpublished one
This sourceGoogle's error-budget construction, in which an availability objective leaves a complementary allowance of downtime over the measurement window that teams may spend.
Supported in partVerified against the source
- The portable inferences carry too: the non-linear cost of each additional nine, the √N pooling benefit, the substitutability of inventory, capacity, and redundancy buffers for the same level, and the commitment-device effect by which a published threshold reads differently from an unpublished one
Mechanisms¶
- After-Action Coordination Review
- The discipline that guards against all three is to protect blamelessness deliberately and to require that every finding leave the room as an owned commitment with a due date — a lesson without an owner is not an output.
This sourceMakes blamelessness a deliberate postmortem principle by focusing on contributing system causes instead of individual blame.
- The discipline that guards against all three is to protect blamelessness deliberately and to require that every finding leave the room as an owned commitment with a due date — a lesson without an owner is not an output.
- Alert Deduplication and Grouping Rule
- This is the discipline behind mature on-call practice, where alerts are expected to be deduplicated into incidents rather than delivered raw.
This sourceDocuments grouping multiple related alerts into a single incident to control duplication and on-call noise.
- This is the discipline behind mature on-call practice, where alerts are expected to be deduplicated into incidents rather than delivered raw.
- Blue-Green or Canary Replacement
- The discipline that guards against it is to gate promotion on staying inside an explicit error budget,
This sourcePromotion can be gated by the remaining error budget, with canary duration and population sized to bound deployment risk.
- The discipline that guards against it is to gate promotion on staying inside an explicit error budget,
- Canary or Limited Rollout
- Its strength is that it earns real-consequence evidence
This sourceShows that canarying gathers evidence under real traffic while limiting exposure, and that error budgets constrain the production risk available for releases.
- Its strength is that it earns real-consequence evidence
- Capacity Headroom
- It misleads when the true boundary isn't the resource being padded
This sourceWarns that capacity judgments mislead when they monitor the wrong resource because a different resource or dependency may be the binding failure limit.
- It misleads when the true boundary isn't the resource being padded
- Capacity Threshold Alert
- But the queue has a saturation boundary: past a certain depth, waiting requests time out, clients retry, retries pile more work onto an already-full queue, and latency does not rise gently — it goes vertical.
This sourceShows how saturated queues cause missed deadlines and client retries that amplify overload.
- But the queue has a saturation boundary: past a certain depth, waiting requests time out, clients retry, retries pile more work onto an already-full queue, and latency does not rise gently — it goes vertical.
- Collision Incident Playbook
- A blameless post-incident review of each activation feeds fixes back into namespace sizing, closing the loop between response and prevention.
This sourceExplains that incident review should feed preventive action, linking response to prevention.
- A blameless post-incident review of each activation feeds fixes back into namespace sizing, closing the loop between response and prevention.
- Escalation Ladder and Repair Review
- The repair-review loop is what makes it more than a phone tree — a blameless postmortem converts each failure into a durable improvement
This sourceDescribes blameless postmortems as learning reviews that produce preventive follow-up actions instead of punishment or blame.
- The repair-review loop is what makes it more than a phone tree — a blameless postmortem converts each failure into a durable improvement
- Failed-Attempt Recovery Workflow
- The discipline is to make recovery idempotent so retries can't double-serve, cap retries with escalation to human judgment, bound the recipient's burden explicitly, and feed recurring failure causes back upstream rather than absorbing them forever.
This sourceRecommends idempotent automated fixes, bounded retries that stop and surface failure, and postmortem actions that address root causes to reduce recurrence.
- The discipline is to make recovery idempotent so retries can't double-serve, cap retries with escalation to human judgment, bound the recipient's burden explicitly, and feed recurring failure causes back upstream rather than absorbing them forever.
- Feedback Latency Monitor
- When one day the latency jumps to twenty minutes, that's the alarm: feedback is arriving late, and rolling out further changes before the last one has registered risks stacking corrections blind — the same trap the team tracks with its mean-time-to-detect discipline.
This sourceRecommends staged changes and observation before wider rollout to avoid compounding an unmeasured bad change.
- When one day the latency jumps to twenty minutes, that's the alarm: feedback is arriving late, and rolling out further changes before the last one has registered risks stacking corrections blind — the same trap the team tracks with its mean-time-to-detect discipline.
- Forecast Trigger Dashboard
- It is deliberately built around a handful of high-value signals rather than a wall of graphs
This sourceRecommends focusing monitoring on a small set of high-value signals and keeping alerting simple, high-signal, and low-noise.
- It is deliberately built around a handful of high-value signals rather than a wall of graphs
- Handoff Readiness Check
- Formalizing the condition as an explicit objective — the kind of freshness-and-completeness target teams codify as a service-level objective — turns "looks done" into a checkable pass/fail rather than a hopeful glance.
This sourceDefines service-level objectives as measurable targets, recommends latency and correctness indicators for data systems, and uses comparison with those targets to decide whether action is needed.
- Formalizing the condition as an explicit objective — the kind of freshness-and-completeness target teams codify as a service-level objective — turns "looks done" into a checkable pass/fail rather than a hopeful glance.
- Incident Playbook Lookup
- Its strength is speed under pressure: it compresses time-to-first-action, cuts mean-time-to-recovery, and encodes hard-won operational knowledge so a junior responder can act like a veteran at 2 a.m.
This sourceReports that a documented and practiced incident-management strategy reduced mean time to recovery.
- Its strength is speed under pressure: it compresses time-to-first-action, cuts mean-time-to-recovery, and encodes hard-won operational knowledge so a junior responder can act like a veteran at 2 a.m.
- Incremental Hotfix or Patch
- Kept honest, it shortens mean-time-to-recovery without trading away safety.
This sourceConnects rapid incremental repair or rollback with reducing mean time to recovery.
- Kept honest, it shortens mean-time-to-recovery without trading away safety.
- Learning Retrospective
- The session runs in the spirit of a blameless postmortem — the goal is to metabolize what the team learned into how it will work, not to find who to blame for having believed the wrong thing.
This sourceDefines blameless postmortems as learning-focused reviews that turn failure findings into system improvements without assigning individual blame.
- The session runs in the spirit of a blameless postmortem — the goal is to metabolize what the team learned into how it will work, not to find who to blame for having believed the wrong thing.
- Minimum-Service-Floor Dashboard
- The floor behaves like an error budget — a guardrail that says how much degradation is acceptable before action is forced.
This sourceDefines an error budget as the allowed degradation before reliability action is required.
- The floor behaves like an error budget — a guardrail that says how much degradation is acceptable before action is forced.
- Mutation Budget Cap
- Exploration stays vigorous where it is safe and tightly rationed where it is not.
This sourceUses remaining error budget to permit faster change when reliability is healthy and to slow or halt releases when the budget is depleted.
- Exploration stays vigorous where it is safe and tightly rationed where it is not.
- On-Call Rotation Activation
- The discipline that guards against this is treating responder attention as a budgeted, protected resource — page only what a human must act on now, and make relief mandatory rather than optional.
This sourceTreats responder attention as a limited workload and reserves pages for urgent events requiring immediate human action.
- The discipline that guards against this is treating responder attention as a budgeted, protected resource — page only what a human must act on now, and make relief mandatory rather than optional.
- Post-Incident Autoimmune Review
- Modeled as a blameless postmortem
This sourceExplains that blameless postmortems encourage participants to disclose what happened without fear of punishment or self-protective blame.
- Modeled as a blameless postmortem
- Post-Incident Review (Hotwash)
- The output is not a fat report but a short list of concrete changes — fix the frequency-assignment step, add a checklist item — each with an owner
This sourceShows postmortem action items assigned to named owners.
- The output is not a fat report but a short list of concrete changes — fix the frequency-assignment step, add a checklist item — each with an owner
- Postmortem Action Tracking
- Its strength is converting the emotional energy after a failure into durable structural change: it captures the reasoning while it is fresh and forces each lesson to become a commitment someone owns, which is what separates an organization that learns from one that merely feels bad and repeats itself.
This sourceTreats postmortems as a means for organizations to learn from failure rather than merely react and repeat it.
- Its strength is converting the emotional energy after a failure into durable structural change: it captures the reasoning while it is fresh and forces each lesson to become a commitment someone owns, which is what separates an organization that learns from one that merely feels bad and repeats itself.
- Recurring Incident Prevention
- The discipline of an honest, blameless review of each recurrence is what makes the tracking trustworthy
This sourceExplains that blameless postmortems reduce fear of punishment, encourage issues to be surfaced, and support trustworthy organizational learning.
- The discipline of an honest, blameless review of each recurrence is what makes the tracking trustworthy
- Redundant Server
- The classic misuse is a redundant server placed in the same rack on the same power feed — duplication that provides no independence, and quietly fails exactly when the primary does.
This sourceTreats a rack and its shared power supply as one failure domain, so colocated replicas do not provide independent protection against that failure.
- The classic misuse is a redundant server placed in the same rack on the same power feed — duplication that provides no independence, and quietly fails exactly when the primary does.
- Release Staggering
- The guarding discipline is to give the shared absorbers an explicit error budget
This sourceDefines an error budget as the allowed unreliability available to govern release activity.
- The guarding discipline is to give the shared absorbers an explicit error budget
- Replica-Lag and Freshness Dashboard
- Its strength is making an abstract guarantee concrete and accountable
This sourceDefines a service-level objective as an explicit target value or range for a measured service level.
- Its strength is making an abstract guarantee concrete and accountable
- Retrospective Action-Item Loop
- When it also separates observation from blame — treating a failure as a property of the system rather than a person — people surface the honest signal instead of hiding it, the core of a blameless postmortem.
This sourceDefines blameless postmortems as system-focused investigations that avoid individual blame so people surface incidents and contributing causes.
- When it also separates observation from blame — treating a failure as a property of the system rather than a person — people surface the honest signal instead of hiding it, the core of a blameless postmortem.
- Retrospective Period Labeling Session
- It draws on the "blameless" retrospective tradition, where the discipline of naming phases for learning rather than fault is what makes honest disclosure possible.
This sourceExplains that blame and shaming deter people from bringing issues to light, whereas a blameless postmortem gives people confidence to disclose and escalate problems.
- It draws on the "blameless" retrospective tradition, where the discipline of naming phases for learning rather than fault is what makes honest disclosure possible.
- Reversible Transformation Sandbox
- Rollback Checkpoint
- The checkpoint turned a risky change into a reversible experiment.
This sourceDescribes a canary as a partial, time-limited deployment that is evaluated before continuation and can be quickly rolled back to a known-good state.
- The checkpoint turned a risky change into a reversible experiment.
- Root-Cause Analysis Handoff
- When that condition holds, ownership passes to reliability engineering, who run a proper root-cause analysis off the critical path.
This sourcePlaces service restoration ahead of root-cause analysis, preserving evidence so diagnosis can continue after the immediate incident is stabilized.
- When that condition holds, ownership passes to reliability engineering, who run a proper root-cause analysis off the critical path.
- Runbook
- Pairing it with a blameless postmortem after each real use is how the runbook keeps improving: incidents feed corrections back into the steps and branches rather than into blame.
This sourceUses blameless postmortems to turn incident learning into preventive actions and system improvements rather than individual blame.
- Pairing it with a blameless postmortem after each real use is how the runbook keeps improving: incidents feed corrections back into the steps and branches rather than into blame.
- Service-Level Definition
- It names the service-level indicator (SLI
This sourceDefines an SLI as a quantitative service measure and an SLO as a target value evaluated over a measurement window.
- It names the service-level indicator (SLI
- Service-Level Tolerance
- The complement — 0.1% of the month, about 43 minutes — is the error budget: the amount of unavailability the service may spend before the tolerance is breached.
This sourceDefines an error budget as the permitted unreliability remaining after subtracting an availability objective from perfect service.
- The complement — 0.1% of the month, about 43 minutes — is the error budget: the amount of unavailability the service may spend before the tolerance is breached.
- Staged Absorption Gate
- The kind of phased exposure it formalises is the same logic as a canary release,
This sourceDescribes gradual canary rollout through staged production exposure and verification before broader release.
- The kind of phased exposure it formalises is the same logic as a canary release,
- Staged Rollout
- Its strength is bounded risk with earned confidence: by advancing only on evidence and halting on trouble, a staged rollout keeps the blast radius of any problem to the cohorts already crossed and turns a risky big-bang launch into a series of small, reversible bets.
This sourceDescribes canarying a small production cohort, using test and rollout evidence to expand in stages, and rolling back releases that demonstrate problems.
- Its strength is bounded risk with earned confidence: by advancing only on evidence and halting on trouble, a staged rollout keeps the blast radius of any problem to the cohorts already crossed and turns a risky big-bang launch into a series of small, reversible bets.
- Structural Filter Postmortem
- Substrate Capacity Budget
- Substrate Integrity Dashboard
- This is why a display keyed to error budgets and explicit thresholds is only as good as the objectives behind it.
This sourceExplains that error budgets derive from service-level objectives, so their operational meaning depends on the objectives and metrics chosen.
- This is why a display keyed to error budgets and explicit thresholds is only as good as the objectives behind it.
- Support Line Postmortem
- Done well it follows the blameless postmortem discipline: examine the systemic conditions rather than assign individual fault, so people tell the truth about what happened.
This sourceExplains that blameless postmortems examine systemic causes and create the safety needed for candid accounts of failure.
- Done well it follows the blameless postmortem discipline: examine the systemic conditions rather than assign individual fault, so people tell the truth about what happened.
- Synthetic Probe
- Its strength is proactive, outside-in detection: it finds the broken path before a real user does and reports the experience from where users actually are, catching failures that internal signals miss entirely.
This sourceDefines black-box monitoring from the user’s perspective and recommends synthetic probes that detect end-to-end failures before users report them.
- Its strength is proactive, outside-in detection: it finds the broken path before a real user does and reports the experience from where users actually are, catching failures that internal signals miss entirely.
- Tail Incident Review
- The classic misuse is the blame-driven postmortem
This sourceWarns that blame-focused postmortems suppress learning and recommends identifying systemic causes and preventive actions.
- The classic misuse is the blame-driven postmortem
- Telemetry Proxy Monitoring
- A common framing of the minimum useful set is the four golden signals — latency, traffic, errors, and saturation.
This sourceDefines the four golden signals of monitoring as latency, traffic, errors, and saturation.
- A common framing of the minimum useful set is the four golden signals — latency, traffic, errors, and saturation.
- Weakest-Link Postmortem
- The discipline is deliberately blameless
This sourceA blameless postmortem seeks the missing system conditions and contributing causes rather than assigning fault to an individual.
- The discipline is deliberately blameless
- Workload Rebalancing Routine
- The load it fights is toil — the repetitive, load-bearing operational work that silently accretes onto a few.
This sourceDefines toil as manual, repetitive operational work tied to running a production service.
- The load it fights is toil — the repetitive, load-bearing operational work that silently accretes onto a few.
Verification¶
Does it exist? Not checked yet. This entry carries no identifier to resolve. It was extracted from the citation as written in the article, normalized, and deduplicated against the rest of the registry.
Does it back the claim? Read against the text for 7 of 80 citations: 7 supported in part. Each verdict is shown under its citation below, with what in the work backs the sentence.
Was it audited? Yes. A second, independent pass read the citation against the article text and recorded a verdict.
Support is checked per citation rather than per work — the same source can be cited soundly in one article and wrongly in another. Per-citation recording began recently, so a citation with no recorded check is a gap in the record rather than evidence it went unchecked.
See how references were verified.
Links previously used in the corpus¶
Before the registry existed this work was also linked 9 other ways.
- https://research.google/pubs/site-reliability-engineering-how-google-runs-production-systems/ ×10
- https://sre.google/books/ ×8
- https://sre.google/sre-book/postmortem-culture/ ×6
- https://www.oreilly.com/library/view/site-reliability-engineering/9781491929117/ ×4
- https://sre.google/sre-book/embracing-risk/ ×2
- https://sre.google/sre-book/monitoring-distributed-systems/ ×2
- https://sre.google/sre-book/addressing-cascading-failures/ ×1
- https://sre.google/sre-book/preface/ ×1
- https://sre.google/sre-book/release-engineering/ ×1
Registry ID ref:9ce885604a11 · see in the full table