Fault Tolerant Operation

Keep operating despite partial failure by detecting, isolating, masking, bypassing, or compensating for failed components.

Solution archetype #

439

Problem family

Fragility, Failure & Continuity Risk

Problem subfamily

Fault Containment & Bounded Service Loss

The Diagnostic Story

Symptom: One component fails and the whole system goes down. Operators know something broke but can't tell what or where until users are already affected. Local faults leak into shared state, corrupting data or triggering cascades far from the original failure. The backup exists, but nobody is sure what happens operationally when primary fails.

Pivot: Define the critical function separately from the components that implement it, add detection and isolation so faults can be contained, choose a bounded continuation mode that preserves the critical function within explicit limits, and specify recovery or escalation so the continuation does not become indefinite hidden degradation.

Resolution: Local faults no longer automatically cause global functional collapse. The critical function continues within stated bounds during partial failure. Operators have clearer signals and procedures rather than improvising under stress, and shared state remains reconcilable during and after faults.

Reach for this when you hear…

[on-call SRE] “The database replica went down and took the entire checkout flow with it — we need circuit breaking so one unhealthy dependency doesn't cascade into a full site outage.”

[aviation] “If a hydraulic system fails, I need to know right now what the aircraft can and cannot do — not improvise it under pressure at altitude.”

[hospital operations] “We lost the electronic records system but the nurses kept treating patients because we'd rehearsed the downtime procedure — that's fault tolerance working.”

When This Archetype Applies

Partial catalog groundingSome structural conditions are represented by existing abstractions, but no sufficient condition set is fully represented.

Diagnostic problem

A system must continue a critical function even when some component, input, actor, dependency, pathway, or subsystem fails during operation.

What this problem means

The structural problem is local failure causing global collapse. A component, actor, input, sensor, link, process step, or service can fail during live operation, and the larger system lacks a designed way to keep the critical function separate from that failure.

This problem often appears as brittle coupling. One bad node disables a service; one missing approval blocks urgent work; one corrupted data source contaminates decisions; one failed route halts all delivery; one unavailable system forces staff to improvise. The system may look reliable in nominal conditions, but it has no explicit runtime posture for partial failure.

Show the applicability expression

Applicability expression5 distinct conditions

Live partial failureandSeparate function from componentandExplicit continuation boundsandFault containmentandContinuity-preserving recovery

Algebraic1∧2∧3∧4∧5

groundedpartly groundedopen

5 conditions, all required.

5Required in every casenumbered 1–5

These hold no matter which pattern applies.

1

Live partial failure · grounded

Partial component, input, actor, dependency, or subsystem failure is plausible during live operation. more

and — also required

2

Separate function from component · grounded

The failed part is distinguishable from the service or function that must be preserved. more

and — also required

3

Explicit continuation bounds · grounded

The continuation mode and tolerated fault bound can be stated explicitly. more

and — also required

4

Fault containment · open

Fault effects must be prevented from cascading beyond the failed part. more

and — also required

5

Continuity-preserving recovery · open

Recovery or repair must occur without losing continuity. more

Other requirements and context (1)

Why these sit outside the expression

Application gate — it governs whether applying the archetype is appropriate or material, rather than defining the structural problem itself.

• Application gateStopping the whole system is more costly than tolerating the local fault. more

Coverage

3 of 5 conditions grounded · 2 open.

Read the methodology · Download the trigger-logic data

Mechanisms / Implementations

• Fault Detection and Diagnosis: Makes a local failure observable and names it — sensing that something is wrong and classifying which component failed and how, so the right continuation response can be chosen.
• Fault Isolation: Draws a boundary around a faulty element so its damage cannot spread to healthy parts, then readmits it only once it is verified sound.
• Error Correction: Masks corruption by adding structured redundancy to a single data stream so that a bounded number of errors can be detected and reconstructed to the correct value in place.
• Redundant Voting: Runs the same job on multiple independent replicas and trusts the majority, so a minority of faulty units is outvoted rather than obeyed.
• Bypass Routing: Keeps a critical flow moving by sending work, traffic, or authority along an alternate path around the failed element instead of through it.
• Degraded Operation Mode: Preserves the most important function at deliberately reduced capacity, precision, feature scope, or automation when the full-service posture can no longer be sustained.
• Self-Healing Repair Loop: Closes the loop on a fault autonomously — detect, apply a bounded repair, verify recovery, and reintegrate — restoring capacity without waiting for a human.
• Manual Continuity Workaround: Keeps a critical function going through a rehearsed human procedure when the normal automated path fails, then reconciles the manual work back into the system on recovery.
• Service Continuity Runbook: An operational document that names the protected function and choreographs the whole fault response — detection, isolation, continuation, escalation, and recovery — with roles and authority made explicit.

Related Abstractions

Abstractions this archetype builds on — directly (a source ingredient) or as a related pattern. Links follow the typed catalog namespace.

Built directly on (3)

• Fault Tolerance: Continue operating under failure.
• Resilience: Absorb shocks and adapt.
• Robustness: Maintain functionality under stress.

Also references 10 related abstractions

• Boundary: Defines system limits.
• Continuity: Smooth change without jumps.
• Controllability: Ability to steer system.
• Coupling: Interdependence among subsystems.
• Fail-Safe: Default to safe state on failure.
• Feedback: Outputs influence inputs.
• Functional Redundancy (Degeneracy): Multiple pathways fulfill same function.
• Modularity: Breaks systems into smaller units.
• Observability: Infer internal state externally.
• Redundancy: Duplicate critical components.

Variants

Narrower or domain-specific specializations that share this archetype's core structure. Recognized variants are established; candidate variants are provisional.

Error-Masking Operation · mechanism family variant · recognized

Continue operation by detecting local errors and masking, correcting, or outvoting them before they affect the protected function.

Fault Containment Continuation · risk or failure variant · recognized

Continue the protected function by isolating the failed portion and allowing unaffected portions to keep operating.

Bypass Continuation · implementation variant · recognized

Keep the critical function active by routing work, flow, authority, or information around a failed element.

Self-Healing Operation · implementation variant · recognized

Use an internal repair loop to detect faults, apply corrective actions, verify recovery, and continue operation with minimal outside intervention.

Degraded Continuation Mode · risk or failure variant · merge review

Continue only the most important parts of the function when full-quality operation is impossible after a fault.

Fault-Decoupled Passive Reset · passive recovery variant · recognized

Let an overload passively decouple an obstructing drive so stored energy can restore a safe state, then automatically recouple after the overload clears.

Editorial Notes

Problem Classification

Classification: Fragility, Failure & Continuity Risk → Fault Containment & Bounded Service Loss

Problem kernel: component failure can remove the entire critical function

Rationale: Operation lacks isolation, alternates, and degraded modes that preserve essential service when one dependency fails.

Independent corroboration: The earliest necessary condition in the frozen evidence is: A system must continue a critical function even when some component, input, actor, dependency, pathway, or subsystem fails during operation. That is a fault containment and bounded service loss problem because A local fault or partner failure can eliminate the whole function because failure domains, alternate activation, and essential degraded service are absent.

Review outcome: Independent reviewer agreement; high confidence.