Skip to content

Process Kill or Restart

Recovery method — instantiates Deadlock Resolution

Terminates or restarts one whole participant so every resource it was holding is released at once, breaking the cycle bluntly and rescheduling its lost work afterward.

When a claim can't be cleanly pulled but the deadlock must go, the crude-but-reliable move is to remove a whole participant. Process Kill or Restart terminates (or restarts) one process, service, or job in the cycle, causing all of its held resources to drop simultaneously. Its defining move is that it operates at the granularity of the entire participant, not one claim — which is exactly why it always works to break the loop and exactly why it is expensive: the killed participant's in-flight, uncommitted work is discarded and must be redone. Where Lock Preemption removes a scalpel's worth, this removes the whole hand; the trade is guaranteed release against thrown-away progress.

Example

Two microservices in a request-scoped workflow have gotten into a synchronous cross-call. Service checkout is holding a worker thread and a row-lease while blocked on a call into pricing; pricing is holding its own worker and blocked on a callback into checkout. Both thread pools saturate as more requests arrive. There is no single lock to hand over — the holds are threads, leases, and open connections tangled together — so the orchestrator kills the checkout instance. Every resource it held drops at once, pricing unblocks and drains, and the pool recovers.

The recovery discipline is in what follows the kill. The requests that checkout was mid-processing are not silently lost: they were enqueued idempotently, so a freshly restarted checkout instance rejoins the pool and the orchestrator replays those requests against it. The break is total and instantaneous; the cost — a few hundred milliseconds of redone work — is paid deliberately and made recoverable by rescheduling, not by pretending nothing was dropped.

How it works

Its distinguishing property is coarseness — it doesn't reason about individual claims:

  • Pick the participant to remove. From the cycle, choose the actor whose termination frees the loop at least total cost — usually the one holding the least committed work, or the cheapest to restart.
  • Terminate and release en masse. Kill (or restart) it; the runtime reclaims all its locks, leases, threads, and handles together. No per-resource negotiation.
  • Reschedule the discarded work. The recovery path re-queues whatever the victim was doing so the work is redone rather than lost — the step that separates a clean recovery from data loss.
  • Readmit a fresh participant. A restarted instance re-enters the system under a re-entry rule, ideally with backoff or altered ordering so it doesn't immediately re-form the same cycle.

Tuning parameters

  • Kill vs. restart — terminate and leave the slot empty, or terminate and relaunch. Restart restores capacity but risks looping straight back into the deadlock; a bare kill sheds load but shrinks the pool.
  • Victim selection — least-work-lost, lowest-priority, or youngest participant. This is the same victim-choice dial deadlock recovery always faces, and biasing it consistently at one actor causes starvation.[1]
  • Work-preservation level — from "fire and forget" to fully idempotent replay of the victim's in-flight tasks. More preservation means more machinery (queues, idempotency keys) but less lost work.
  • Re-admission backoff — how long and with what altered ordering a restarted participant waits before rejoining, tuned to keep it from re-deadlocking on entry.

When it helps, and when it misleads

Its strength is dependability and simplicity: dropping an entire participant is guaranteed to break any cycle it sits on, needs no fine-grained lock surgery, and is often the fastest way back to a running system when the holds are too entangled to unpick. For stateless or idempotent participants the cost is genuinely small.

It misleads whenever the victim's work is not cheap to redo. Kill a participant holding uncommitted, non-idempotent state and you trade a deadlock for lost or corrupted work — the failure mode the archetype warns about. Restarting without changing anything can produce a livelock of kill-restart-redeadlock. And it is a technical recovery only: you cannot "kill and restart" a stalled contract negotiation or a frozen approval. The discipline is to make the victim's work recoverable before you rely on killing it — enqueue idempotently, checkpoint, or accept the loss explicitly — and to alter ordering on restart so the cycle can't immediately re-form.

How it implements the components

Process Kill or Restart fills the blunt-break-and-recover components — act on the whole actor, then make its loss good:

  • bounded_break_action — the intervention is terminating one participant; bounded to a single actor even though it releases all that actor's holds.
  • recovery_path — the discarded in-flight work is rescheduled or replayed so termination doesn't become silent data loss.
  • participant_reentry_rule — a restarted instance is readmitted under controlled ordering/backoff so it resumes service without re-deadlocking.

It does not preserve the victim's prior consistent state (safe_state_checkpoint) the way Rollback to Safe State does — it discards and redoes — nor does it choose a single claim to sever (break_point_selection), which is Lock Preemption's finer move; and it carries no external legitimacy (resolver_authority).

References

[1] Victim selection is the standard term in operating-system deadlock recovery for choosing which process to abort or roll back. The recurring hazard is that a fixed rule (e.g. "always kill the youngest") can repeatedly victimize the same process, converting deadlock recovery into starvation — which is why victim rules are usually aged or rotated. ↩