Skip to content

Projection-Frontier Dashboard

A monitor — instantiates Event-Log-Centered Modeling

Shows how far each projection has consumed the log, turning invisible replication lag and coverage gaps into watched, actionable numbers.

In an event-log model every derived view is always somewhere behind the log — it has processed events up to some point and not yet the rest. That point is the projection's frontier, and when it is invisible, a stale dashboard, a lagging search index, or a report missing yesterday's events all look exactly like healthy ones. The Projection-Frontier Dashboard makes the frontier visible: for each projection it shows how far through the log it has consumed, how far that is behind the log's head, and therefore how much you can trust what it's currently showing. Its distinguishing job is not to compute a view or check its correctness but to answer a freshness question the views themselves cannot ask about themselves — "as of what point in history is this answer true, and what is it still missing?"

Example

A product-analytics platform serves a dozen dashboards, each a projection consuming the same clickstream event log. One Monday, revenue numbers look suspiciously flat. The Projection-Frontier Dashboard shows why in one glance: eleven projections sit within seconds of the log head (frontier essentially current), but the revenue projection's frontier is stuck eight hours back — its consumer crashed overnight and stopped advancing. The flat revenue isn't a business signal; it's a projection that simply hasn't seen the morning's events yet.

The same panel carries a completeness line alongside the frontier: the mobile-events source has been flagged as only ~90% delivered for the last hour because an upstream collector is dropping batches. So even the projections that are caught up to the log head are shown as resting on incomplete input. The team now knows precisely what to do — restart the stalled revenue consumer, and treat all of the last hour's mobile figures as provisional — rather than debugging a revenue "drop" that never happened.

How it works

  • Track each projection's frontier. Record the position in the log up to which every projection has processed, and express it both as an absolute point and as distance behind the log's head (the lag).
  • Turn lag into freshness. Translate raw frontier distance into the terms operators reason in — seconds behind, "current as of 09:14," number of unprocessed events — so a number implies an action.
  • Surface coverage, not just position. Carry a completeness signal for the underlying sources, so a projection caught up to the head is still shown as trustworthy-only-as-far-as its input is complete.
  • Alert on stall and drift. Flag frontiers that stop advancing or fall past a threshold, distinguishing a projection that is slow from one that has stopped.

Tuning parameters

  • Lag alert threshold — how far behind counts as a problem. Tight thresholds catch stalls instantly but cry wolf during normal bursts; loose thresholds are calm but let real staleness build.
  • Frontier unit — whether lag is shown in events, in time, or in business terms ("as of last complete hour"). The unit decides who can act on it — engineers read events, stakeholders read time.
  • Completeness sensitivity — how aggressively under-delivery is flagged. High sensitivity surfaces every dropped batch and risks alarm fatigue; low sensitivity keeps the board green and can hide real gaps.
  • Per-projection weighting — which projections' freshness is treated as critical versus informational. Focus attention on the views that drive decisions; demote the ones that can safely lag.

When it helps, and when it misleads

Its value is turning an invisible property — how current and how complete is this derived view — into a watched number, so a stalled consumer or a dropping source is caught in seconds instead of discovered through a mystified user. It is the operational nerve of a projection-heavy model: without it, "the dashboard is wrong" and "the dashboard is merely behind" are indistinguishable.

The dashboard's own trap is that a frontier is a claim about position, not correctness. A projection can be perfectly caught up to the log head and still be wrong — a projector bug produces a current-but-incorrect view that the frontier reports as green.[1] Worse, a frontier that has advanced past a problematic event can hide that the event was skipped rather than handled, so "caught up" masks a silent drop. And a completeness signal is only as honest as the upstream system's willingness to admit it is missing data. The disciplines: read the frontier strictly as freshness and pair it with Projection Rebuild and Diff for correctness; treat "caught up" as "has consumed," not "has consumed correctly"; and prefer completeness signals sourced from expected-versus-received counts rather than the projection's own optimism.

How it implements the components

The Projection-Frontier Dashboard realizes the archetype's observability slice — how current and how complete each derived view is — and none of the derivation, storage, or correctness slices:

  • projection_frontier — its core readout: the per-projection consumed-position and its lag behind the log head, made continuously visible.
  • event_completeness_statement — it surfaces, alongside each frontier, how complete the underlying capture is, so freshness is never mistaken for full coverage.

It monitors but does not set the rule for handling stragglers — the late-and-out-of-order-event policy it visualizes is owned by Provenance-Weighted Event Reconciliation — and it does not verify projection correctness; that is Projection Rebuild and Diff.

References

[1] In stream processing a watermark is the system's assertion that it has seen all events up to a given time, letting downstream steps decide when a window is safe to close. A frontier is the same idea applied to a projection — a statement of progress through the log — and shares the watermark's caveat: it certifies how far, never how correctly. ↩