Skip to content

Perturb-and-Relock Drill

Validation process — instantiates Decentralized Phase Locking

Deliberately shocks an already-locked population and measures how fully and how fast it re-locks, before real dependence is placed on the lock.

A population that locks under calm conditions may shatter under a jolt — and you would rather learn that in rehearsal than in production. Perturb-and-Relock Drill deliberately injects a controlled disturbance into an already-locked population — a phase kick, a detuned unit, a dropped link — and measures how completely and how quickly the lock returns. Its defining trait is that it is proactive and adversarial to itself: it manufactures the failure on purpose, on a schedule, before anyone depends on the lock, and it reads recovery not from a single global number but from the fine grain, so a "recovered" average that hides a stranded minority gets caught. It is a test harness, not a recovery maneuver — it breaks things to measure, and leaves the fixing to other mechanisms.

Example

A distributed database keeps its replicas loosely phase-locked so their disk-heavy compaction cycles never all fire at once. Before trusting that in production, the team drills it: inject a 200 ms delay into one replica's timing channel, watch the coherence panel until phase error falls back under threshold — then, separately, sever a link for 30 s and watch again. The population re-locks in ≈40 s after the delay. But the severed link leaves two replicas settled into their own phase island that never rejoins the rest — a chimera the global order parameter, sitting near "mostly locked," never revealed. The cluster monitor is what surfaces it, and the team fixes the fragility before it becomes a 3 a.m. incident.

How it works

What distinguishes the drill from passive monitoring or a live recovery:

  • A menu of deliberate faults. Phase kicks, frequency detuning, link loss, unit restarts — applied on purpose and repeatably, not waited for.
  • Recovery is measured, not assumed. Each fault is followed by reading time-to-relock, residual phase error, and slip count off the panel.
  • The fine grain is watched, not just the average. A cluster/outlier monitor runs alongside so partial recovery — a stranded island — can't hide behind a healthy-looking global metric.

Tuning parameters

  • Perturbation menu and severity — big rare shocks probe worst-case; small frequent ones probe steady resilience.
  • Single vs compound faults — compound faults (delay and link loss) expose interaction failures but are harder to attribute.
  • Recovery threshold and timeout — how close to target counts as "recovered," and how long you wait before calling it a failure.
  • Observation granularity — global order parameter versus per-cluster; the coarser view can mask a stranded minority.
  • Blast radius / safety bound — the cap that keeps a drill from becoming a real outage.

When it helps, and when it misleads

Its strength is buying evidence early: it exposes overshoot, slow recovery, and hidden cluster or chimera states before operational dependence grows, when they are cheap to fix.

It misleads if its passes are over-read. A drill only tests the faults you thought to inject; surviving them proves resilience to those, not to the novel shock that actually takes you down. And a global order parameter can report "recovered" while a minority stays stranded — the reason the monitor is not optional. The classic misuse is the token drill run to certify readiness — theater that exercises only the easy faults.[1] The discipline is to escalate severity, always watch the fine grain and not just the average, and treat any pass as bounded evidence rather than proof.

How it implements the components

  • perturbation_and_recovery_test — the drill is this: a deliberate shock followed by measured recovery, repeated across a fault menu.
  • coherence_and_order_parameter_panel — during each shock it reads recovery time, residual phase error, and slips from the panel to score how fully the lock returned. It uses the panel in the transient; continuous steady-state monitoring is local_coherence_probe's.
  • cluster_chimera_and_outlier_monitor — it watches the monitor to catch partial recovery that the global average hides; continuous scanning is cluster_and_chimera_scan's, while the drill invokes it under perturbation.

It does not perform the live recovery maneuver — that is Phase-Slip Recovery Protocol — nor find the cold-start capture threshold, which is Weak-Coupling Ramp Trial.

Notes

Keep the drill distinct from Phase-Slip Recovery Protocol: the drill is a proactive rehearsal that injects a fault to measure recovery; the protocol is the reactive maneuver that performs the recovery when a slip actually happens in operation. The drill tests whether that protocol is good enough.

References

[1] Chaos engineering is the practice of deliberately injecting failures into a running system to reveal weaknesses before they cause real outages. This drill applies that stance to phase locking — the honest version tests worsening, novel faults, not a fixed rehearsed few. ↩