Skip to content

Projection Rebuild and Diff

An audit — instantiates Event-Log-Centered Modeling

Rebuilds a projection from the log and diffs it against the live view, treating any disagreement as evidence the view is wrong, never the log.

In an event-log model the derived views are supposed to be replaceable functions of the log — but live projections drift: a projector bug drops a case, a hot-fix is applied to the materialized table by hand, an event type is mishandled, and the view quietly stops matching the history it claims to summarize. Projection Rebuild and Diff is the audit that catches this. It rebuilds a projection from scratch off the authoritative log and diffs that clean rebuild against the currently-served view; every discrepancy is a defect. Its defining commitment — the thing that makes it this mechanism and not a generic data-quality check — is directional trust: the log is the source of truth, so when the rebuild and the live view disagree, the verdict is always that the view is wrong, and the log is what adjudicates.

Example

A distribution warehouse serves "units on hand" from a materialized inventory table that a projector updates as movement events stream in — receipts, picks, returns, cycle-count adjustments. Physical counts have started drifting from the system by a few units per SKU, and nobody knows whether the floor is miscounting or the software is. Projection Rebuild and Diff settles it: overnight it replays the entire movement log into a fresh, throwaway inventory projection and diffs it against the live table. The diff is decisive — 47 SKUs disagree, all in the same direction, and every one had a return-to-vendor event in the last quarter. The live projector, it turns out, never decremented on that event type.

The rebuild matched the physical counts; the live table did not. Because the log is treated as authoritative, the finding is unambiguous — the projector is buggy, not the events — and the fix is to correct the projector and re-materialize, not to patch the table. The same diff, run as a nightly job, becomes an early-warning tripwire: any day the rebuild and the live view diverge, something has gone wrong in derivation before customers ever see a wrong number.

How it works

  • Rebuild from the source of truth. Derive the projection again, cleanly, from the log alone — no reuse of the live view's current state — so the rebuild is an independent second opinion grounded in the authoritative history.
  • Diff against the live view. Compare the fresh rebuild to the currently-served projection key-by-key, producing an itemized list of disagreements rather than a single pass/fail.
  • Attribute every discrepancy to the view. Under the model's directional trust, a mismatch means the live view diverged from the log — from a projector bug, an out-of-band edit, or a mishandled event — never that the log is wrong.
  • Localize the cause. Cluster the discrepancies (which keys, which event types, which time window) so the diff points at the specific derivation defect, not just its existence.

Tuning parameters

  • Diff granularity — whole-view checksum, per-key comparison, or field-level. Coarse diffs are cheap and only say that something is off; fine diffs cost more and say what.
  • Scope of rebuild — full-history rebuild versus a bounded recent window. Full rebuilds are authoritative but expensive; windowed rebuilds are cheap tripwires that can miss old, slow drift.
  • Tolerance band — whether tiny, benign differences (rounding, in-flight events) are ignored or flagged. Too tight and every run is noisy with false alarms near the frontier; too loose and real drift hides under the threshold.
  • Cadence — on-demand forensics versus a scheduled guard. Frequent runs catch drift early at ongoing cost; rare runs are cheap but let defects live longer.

When it helps, and when it misleads

Its strength is that it makes projection correctness checkable rather than assumed: it converts "the numbers look off" into a precise, log-grounded list of what disagrees and where, and it doubles as a regression guard that catches derivation bugs before users do. It is the concrete payoff of an event-log model — because state is a rebuildable function of history, correctness has an oracle.

Its sharpest trap is the shared-bug blind spot: if the rebuild runs the same projector code as the live view, the two agree perfectly while both are wrong — the diff certifies a bug instead of catching it.[1] Non-determinism is the other noise source: if rebuilding the projection isn't reproducible, diffs flicker with meaningless differences and real ones drown. And a diff run only near the live frontier will show spurious mismatches from events still in flight. The disciplines: rebuild with an independent or explicitly-audited derivation path rather than blindly reusing production code, require deterministic replay so a clean rebuild is truly reproducible, and diff up to a settled point in the log rather than the moving edge.

How it implements the components

Projection Rebuild and Diff realizes the archetype's verification slice — proving derived views still equal the history they claim — and none of the storage, timing, or content-modelling slices:

  • projection_reconciliation_check — it is this check: the rebuild-and-compare that reconciles a live projection against the log and itemizes every divergence.
  • source_of_truth_reference — its directional trust operationalizes the log as source of truth; the rebuild is grounded in the log, and the diff's verdict always falls on the view.

It does not itself provide the mechanism that replays and rebuilds efficiently — it consumes that from Snapshot Plus Replay and depends on a deterministic path from Deterministic Replay Protocol — and it does not define what the projection is; that contract belongs to Event-Sourced Projection.

Notes

The audit checks that a projection is faithful to the log; it says nothing about whether the log's events are themselves correct. Contested or conflicting source events are a different problem, handled upstream by Provenance-Weighted Event Reconciliation. Keeping the two apart matters: a clean diff certifies faithful derivation, not true history.

References

[1] Reconciliation — independently recomputing a figure from source records and comparing it to the reported one — is a control borrowed from accounting. Its cardinal rule is independence: a reconciliation that reuses the very process it is meant to check provides no assurance, which is why a rebuild must not silently inherit the live projector's bug. ↩