# Experiment d25: System Migration with Hidden Dependency Protection

## Working

**Problem:** Regional utility system migration with undocumented downstream dependencies. Shutdown is irreversible once hardware reclaimed. Design needed to prevent hidden failures from causing sustained outages.

**Verbs Run (order and why):**
1. **MATCH** — Anchored problem to reversibility_horizon (time window before lock-in), fault_tolerance (system must detect and respond to failures), irreversibility (hardware reclamation point), FMEA (discover hidden paths), layering (stage transitions), fading (controlled removal). Used search_prime (domain-stripped meta-model) and search_by_facets (multi-structure problem).
2. **SALIENCE-RANK** — Reversibility_horizon is load-bearing: defines the window between cutover and hardware reclamation (point of no return). Fault_tolerance is load-bearing: requires explicit redundancy and monitoring rather than hoping nothing breaks.
3. **COMPOSE** — Three archetype pillars: checkpoint_and_rollback (preserve old system as fallback), dependency_exposure (discover hidden callers), coordination_across_reentry (phase the transition with gates).
4. **LIFT** — Abstracted domain specifics: irreversible commitment on incomplete information; time window for cheap undo; fault model of hidden coupling; mitigation through exposure, staging, abort capacity.
5. **TRANSPORT** — Mapped checkpoint_and_rollback roles: old system = known-good state; configs+snapshot = checkpoint; disable-old/enable-new = risky change; service disruption = rollback trigger; restore-from-snapshot = restoration path.
6. **EVALUATE-FIT** — Checked anti-signatures: change is not inherently irreversible if hardware retained (extends horizon); environment changes moderately (mitigate via continuous monitoring); cost-benefit favors hardware retention over outage risk. **Verdict: FITS.**
7. **RECONCILE** — Integrated three archetypes: pre-cutover (expose dependencies via FMEA, logs, interviews), parallel running (preserve reversibility via old system as live backup), staged transition (phase traffic with abort gates at each boundary), post-cutover (fade old system gradually).

**COST:** ~8 encyclopedia operations; search_by_facets used; 7 verbs run; recommendation 947 words; no errors.

---

===RECOMMENDATION_BEGIN===

## Design for Safe System Migration with Hidden Dependency Protection

The migration strategy centers on three commitments: (1) expose hidden dependencies before the point of no return, (2) preserve the ability to abort and rollback throughout the transition, and (3) phase the cutover in small increments with explicit abort criteria at each phase.

### Pre-Cutover Phase: Dependency Exposure and Inventory

Begin with a 60–90 day intensive discovery period before any production traffic shifts to the new system. This is not optional and cannot be rushed.

Use multiple discovery methods in parallel. Deploy network monitoring and log analysis to capture all connections to the old system—not just the documented ones. Many utilities find that 20–40% of dependencies are undocumented scripts, ad-hoc reports, third-party integrations, or workarounds built by technicians. Interview every team that touches operations, billing, customer service, or field dispatch. Ask not "what systems do you use?" but "what would stop working if we turned this off?" Conduct a FMEA focused on the old system: for each critical function, enumerate how downstream processes would break if that function failed. Review contracts and SLAs with other utilities, municipal systems, or regulatory agencies that may depend on data feeds or services from the old system.

For each discovered dependency, record four things: (1) the downstream tool or process, (2) the old system function it depends on, (3) criticality (would an outage of this dependency cause operational disruption within minutes, hours, or days), and (4) the person or team responsible for that downstream system. Unknown dependencies are treated as high-criticality until proven otherwise.

Create a dependency matrix. Rows are downstream tools; columns are old system functions. Cells indicate criticality. Review this matrix weekly with operations teams to catch omissions. The matrix is the control document for the entire migration; it is more important than the project schedule.

### Parallel Running Phase: Extending the Reversibility Horizon

After dependency inventory, bring the new system into production in a parallel configuration: both systems run simultaneously, with the old system as the primary and the new system processing a monitored subset of transactions or a read-only replica of the workload.

During parallel running, the reversibility horizon does not start counting down. The old system remains alive and handling real transactions. If catastrophic failure occurs in the new system or hidden dependencies break, the team can shut down the new system and revert to 100% old-system operation with no service interruption. This window typically lasts 6–12 weeks, depending on system stability and confidence.

The critical rule during parallel running: both systems must process the same data and transactions, and their results must be continuously compared. If the new system produces different outputs, escalate immediately. Do not continue to the next phase. Silent divergence is the enemy. Use automated reconciliation to catch output mismatches, transaction counts, data integrity issues, and computation differences. Alert the team within hours of any anomaly.

Maintain a live copy of the old system's data as a checkpoint. If the new system causes a data corruption or introduces a hidden dependency failure, the team can restore from the old system's snapshot. Keep this checkpoint current throughout the parallel window. The cost of storage and compute for this checkpoint is trivial compared to the cost of a week-long outage.

### Staged Traffic Transition: Phased Cutover with Abort Criteria

Once the systems have run in parallel for a stabilization period (typically 4–8 weeks of clean reconciliation), begin migrating traffic in phases:

**Phase 1 (Week 1–2): Canary Cohort**  
Move 5% of transactions to the new system. Monitor error rates, latency, service disruption, and data consistency. Set abort threshold: if error rate exceeds 1% above baseline, or if any critical dependency shows degradation, pause immediately and revert to 100% old system. Do not continue until the cause is understood and fixed.

**Phase 2 (Week 3–4): Early Adopters**  
Move 25% to the new system, including a selected set of critical facilities. Monitor the dependency matrix: is every dependency functioning as expected under load? Set abort threshold: error rate >0.5% above baseline, or latency >200ms above baseline, or any critical dependency owner reports an incident.

**Phase 3 (Week 5–6): Majority**  
Move 75% to the new system. By now, most hidden dependencies should have surfaced and been either fixed or documented. Continue monitoring against the same thresholds.

**Phase 4 (Week 7+): Complete**  
Move 100% to the new system, but keep the old system running in warm-standby mode for 30 days. Any incident that cannot be resolved in the new system can trigger an emergency revert: switch traffic back to the old system (which is already running and current). After 30 days of stable new-system operation, old system hardware can be archived, then decommissioned.

At each phase boundary, a cross-functional gate meeting reviews:
- Dependency inventory status: have all known dependencies proven stable under load?
- Error and latency telemetry: do they remain within acceptable bounds?
- Incident reports: any surprises, workarounds, or unexpected couplings discovered?
- Dependency owner sign-off: has each team responsible for a critical downstream system confirmed readiness for the next phase?

If any gate review surfaces unresolved dependencies or instability, pause the phase and extend the current cohort until the issue is understood.

### Monitoring and Abort Protocol

Throughout the migration, establish a shared monitoring dashboard visible to operations and the dependency owners. The dashboard shows:
- Current traffic distribution between old and new systems
- Error rates in each system and in each major downstream dependency
- Latency percentiles (p50, p95, p99) for old vs new system
- Dependency health status for each critical tool in the inventory
- Phase gates and current approval status

Define explicit abort thresholds:
- If old system latency spikes or error rate rises during new-system operation (indicating coupling or resource contention), pause traffic migration and investigate.
- If any high-criticality dependency shows degradation under new-system operation, pause the current phase and focus on that dependency.
- If a dependency owner reports unplanned incidents or workarounds during a phase, the phase manager can freeze traffic migration until the cause is confirmed.

The abort decision is not escalated upward; it is automatic at threshold. Do not require meetings or permission to pause. The reversibility horizon closes quickly once cutover accelerates. The team's authority to pause is the safety valve.

### Post-Cutover: Fading the Old System

After 30 days of stable new-system-only operation, the old system is still running but carrying no production traffic. Document the lessons: which dependencies were hardest to discover, which failures appeared under load, which processes needed workarounds. Update the dependency inventory for future reference.

Then, rather than abrupt decommissioning, fade the old system gradually. Retain it in cold-standby (no active connections, but hardware preserved and data snapshots current) for 6 months. This provides a final escape route if a hidden dependency emerges weeks after cutover. After 6 months, decommission the hardware.

### Operating Rules

1. **Reversibility is non-negotiable.** Parallel running extends the reversibility horizon to the point where it is no longer economically preferable to revert. Until that point is explicitly reached and signed off, maintain the ability to abort and revert to 100% old-system operation.

2. **Hidden dependencies are discovered through operation, not planning.** The dependency matrix will be incomplete. Assume that real traffic under load will surface surprises. The abort gates protect against those surprises.

3. **Dependency ownership is accountability.** Every critical dependency must have an owner. That owner attends phase gates and signals readiness or concern. Ownership is stewardship, not blame.

4. **Monitoring thresholds are gates.** Do not override a threshold breach with optimism or schedule pressure. A threshold breach is a signal that the system is not ready for the next step.

5. **Communication cadence is continuous.** Weekly status reviews with dependency owners during parallel running, phase-gate reviews at each cutover boundary, and incident reviews for any anomaly. Silence means risk is accumulating.

===RECOMMENDATION_END===
