Data Lineage Balance Check¶

Audit — instantiates Conserved Reservoir-Flux Balancing

Asserts that every step of a data pipeline conserves its records and totals — what enters equals what leaves plus what was intentionally dropped — and flags any hop where the count silently breaks.

A Data Lineage Balance Check treats a data pipeline as a conserved-flow network. At each transformation, the quantity entering — a row count, the sum of a monetary column, a checksum — must equal what leaves plus what was deliberately filtered out, and the check asserts that identity automatically on every run, flagging the exact hop where conservation breaks. The idea that makes it this mechanism: it applies the conservation law to information moving through transforms, per-hop and programmatically, carrying an explicit tolerance band for the drift that legitimately occurs (late-arriving records, rounding). It is a detector — a pass/fail conservation assertion with localization — distinct from the material balance table and from the human reconciliation workflow.

Example¶

A bank's nightly ETL moves transactions from a core banking system into an analytics warehouse through roughly eight stages — extract, dedupe, currency-normalize, enrich, load. A balance check attaches a control total^[1] to the run: the count and summed amount of transactions leaving the source must equal what lands in the warehouse plus any explicitly quarantined rows. One night the summed amount is short by ≈$2.3M. Because the identity is asserted at every hop, the check localizes the break to the currency-normalization step, where a null exchange rate had silently dropped roughly 40 foreign-currency transactions. The break is caught before the warehouse feeds a regulatory report — not a quarter later in an external audit — because conservation was checked at each transform rather than only end-to-end.

How it works¶

Instrument each transform with an input and output measure — count, summed value, or checksum on the records passing through.
Assert conservation per hop: output + intentionally-dropped = input, within tolerance.
Carry a tolerance band for legitimate drift, so late data and rounding do not trip a false break.
On a break, report the specific hop and the size of the gap — localization, not just a red light.

Tuning parameters¶

Balance measure — row count versus summed value versus cryptographic checksum. Value and checksum catch corruptions that preserve row count; a count alone misses them.
Tolerance band — how much drift is allowed before failing. Too tight and it flaps on ordinary late data; too loose and it hides real loss.
Granularity — per-hop versus end-to-end only. Per-hop localizes the break but costs instrumentation at every stage.
Failure action — warn, quarantine the batch, or hard-stop the pipeline before downstream consumers read bad data.

When it helps, and when it misleads¶

Its strength is catching silent data loss at the exact transform that caused it, before it contaminates a downstream report — and, in the value or checksum form, catching corruption that a row-count check would wave through.

It misleads when the tolerance is set to make the check pass rather than to model real drift: a band widened until it stops complaining hides the very loss it exists to catch. A count-only check misses value corruption that leaves the row count intact. And by design it detects the break but says nothing about where the lost records went or how to resolve them. The discipline is to match the balance measure to the failure you actually fear, and to treat a widening tolerance as a finding to investigate, not a knob to turn — handing resolution to the reconciliation workflow and the "where did it go" question to the loss audit.

How it implements the components¶

Data Lineage Balance Check realizes the conservation-assertion side of the machinery — the identity and its tolerance, applied to data:

conservation_balance_equation — states and enforces the in = out + intentionally-dropped identity at each transform.
uncertainty_and_measurement_error_band — the tolerance band that separates legitimate drift from a real conservation break.

It only detects the break; it does not apply the rule that resolves it (balance_reconciliation_rule) — that's Inventory Reconciliation Workflow; it does not trace where the lost records went (leakage_or_loss_sink_register) — that's Loss-Sink Audit; and the tabular, material form of the same balance is Mass-Balance Table.

Instantiates: Conserved Reservoir-Flux Balancing — the balance check is the archetype's automated conservation guard on a flow of information.
Sibling mechanisms: Inventory Reconciliation Workflow · Mass-Balance Table · Material Flow Analysis · Capacity Headroom Alert · Compartment Model · Flow Gate or Valve Rule · Loss-Sink Audit · Reservoir Balance Dashboard · Sankey Flow Map · Stock-and-Flow Diagram · System Dynamics Simulation · Unit Conversion Crosswalk · Water or Resource Budget

References¶

[1] A control total — a count or sum computed at the start of a batch and re-checked at the end to prove nothing was lost or added in between. A staple of batch data processing and accounting; the balance check generalizes it from the two endpoints to every hop of a pipeline. ↩