Skip to content

Mobile-Defect Progress Dashboard

Monitor — instantiates Mobile-Defect Reconfiguration

Renders the moving front live — where the defect is now, how much of the substrate is done behind it, and how much damage it is leaving — so operators can read progress and trouble at a glance.

A reconfiguration in flight is nearly invisible: the front is somewhere in the structure, most units look unremarkable, and harm accrues quietly. The Mobile-Defect Progress Dashboard makes it observable — a live positional map of the substrate coloured by state (done / in-flight / untouched) alongside derived metrics of how far the front has travelled and how much damage it is leaving. Its defining trait is that it only ever watches: it moves nothing and decides nothing. Pure instrumentation, it converts scattered telemetry into a single legible picture of the transition as it happens, so a stalling front or rising harm is seen at the front rather than discovered in the aftermath.

Example

An SRE is rolling a new release across roughly 5,000 servers. The dashboard shows a grid of the fleet: green cells are updated and healthy, a band of amber cells is the front currently updating, grey is not-yet-touched. Across the top sit the derived numbers — 62% complete, current rollout velocity and ETA, and a damage panel: error rate and p99 latency versus the pre-rollout baseline, plus error-budget burn. Midway, one amber batch flips red as its error rate spikes. Because the map shows the front stalling there and the damage panel climbing, the operator sees it before customers report anything and pauses the rollout. The dashboard did not stop the rollout — it made the trouble visible in time for a human to.

How it works

The monitor derives two things from raw unit telemetry and never writes back to the system:

  • A positional state map. Each unit's current state and, from their arrangement, the location and width of the front — the live substrate map.
  • Progress-and-damage metrics. Percent complete, velocity, ETA on the progress side; error/latency/data-diff deltas versus baseline on the damage side.

Threshold-based colouring and alerting draw the eye to the two things that matter — where the front is, and where harm is rising — so the picture is read in seconds, not reconstructed from logs.

Tuning parameters

  • Refresh latency — how fresh the view is. Fresher catches trouble sooner but costs load and amplifies noise.
  • Aggregation grain — per-unit vs. per-batch vs. global. Finer shows the exact front; coarser stays readable at fleet scale.
  • Damage-metric choice — which signals count as harm (errors, latency, data divergence, complaint volume). Pick the wrong proxy and real damage stays off-screen.
  • Alert thresholds — how large a deviation trips a warning. Tight is early but noisy; loose is calm but late.
  • History window — live-only vs. a trailing track of where the front has been, which trades clutter for context.

When it helps, and when it misleads

Its strength is making an otherwise invisible in-flight process legible and shared: a stalling front and mounting damage surface early, and everyone watches the same picture. It misleads whenever the displayed state diverges from the real one. A dashboard is a model, not the territory — it can read all-green during a genuine outage if the damage metric fails to capture the harm users actually feel, because teams instrument what is easy to measure rather than what matters. The classic misuse follows directly: the progress number becomes the target and gets gamed, so operators push "percent done" upward while quietly deferring the hardest units — 95% complete concealing the worst 5%.[1] The discipline is to instrument for user-felt damage, not merely count-complete, and to treat the dashboard as a generator of questions rather than a source of ground truth.

How it implements the components

  • progress_and_damage_metrics — its core product: derived progress (percent, velocity, ETA) and damage indicators (error, latency, divergence versus baseline).
  • substrate_state_map — the live positional view of each unit's state and the front's current location and width.

It only displays these; it does not set the population cap it shows (defect_population_control — Defect-Population Limit Protocol), verify that units behind the front are actually correct (behind_front_state_verification — Residual-Defect Scan), or keep the durable move-by-move ledger (history_and_lineage_record — Rolling State-Migration Workflow).

References

[1] Goodhart's law — once a measure becomes a target, it ceases to be a good measure (after economist Charles Goodhart). Here in its everyday form: press on "percent complete" as the goal and it drifts away from the real state it was meant to reflect, as the hardest units get deferred to keep the number rising. ↩