Skip to content

Data Lineage

Recording, for every data element, the complete sequence of sources, transformations, and movements that produced it — as edges in a queryable dependency graph — so audit questions become forward and backward traversals rather than forensic reconstruction.

Core Idea

Data lineage is the discipline of recording, for every data element, the complete sequence of sources, transformations, and movements that produced its current value — with the agents and times at each step — in a form supporting bidirectional traversal of the resulting dependency graph. Backward traversal reaches all upstream sources; forward traversal reaches all downstream consumers. Its distinguishing commitment is queryable graph topology: provenance recorded as graph edges, not a flat audit log.

Scope of Application

Lives within data engineering, wherever information flows through a directed (near-DAG) graph of transforming processes whose provenance can be recorded as queryable topology.

  • Analytics and data warehousing — the home: raw tables through marts to a dashboard tile.
  • Machine-learning pipelines — which data, code, and hyperparameters produced which model version.
  • Feature stores — which sources and transformations feed each served feature.
  • Regulatory compliance — BCBS 239, GDPR, FDA data-integrity traceable provenance.
  • Scientific reproducibility and cybersecurity forensics — auditing a result's or a compromise's derivation.

Clarity

Naming whether a system has lineage makes legible why some failures are tractable and others archaeology — the difference lies not in the team's skill but in whether provenance was recorded as queryable graph topology. It also sharpens what data quality can mean: not a per-snapshot property of a value in isolation, but a property of a value and its derivation, so trust is read off the soundness of the recorded chain.

Manages Complexity

Without lineage, every audit question is open-ended forensic work scaling with total code volume. Lineage records provenance once as an explicit graph, reducing the whole family of audit questions to three traversal primitives. The cost signature becomes precise: audit difficulty depends on the depth and fan-out of the graph local to the affected node, not on system size — a bad number resolves as a bounded backward walk in minutes.

Abstract Reasoning

Treating the processing DAG as the system model licenses directional moves (forward-reachability for change-impact, backward-traversal for root-cause, reachability for blast-radius), boundary-drawing (tractable-versus-archaeology turns on graph-versus-flat-log; resolution time follows local graph shape), and a further reframe of data quality as a value-and-derivation property, converting trust into a backward-traversal verdict over each node's operation, agent, and refresh time.

Knowledge Transfer

Within data engineering the concept transfers literally — the three traversal primitives, the local-cost signature, and the derivation-grounded quality reframe carry across warehousing, ML pipelines, feature stores, and forensics, with tooling (OpenLineage, Atlas, dbt) converged on one graph model. Beyond it the residue is the parents provenance and traceability — recurring wherever information moves through transforming processes, but straining once the directed-graph picture goes metaphorical (social influence). The pipeline toolchain and name stay home.

Relationships to Other Abstractions

Local relationship map for Data LineageParents appear above the current abstraction, mutual partners to the right, and children below. Node labels state whether each abstraction is prime or domain-specific; colors identify relation types.Data LineageDOMAINPrime abstraction: Traceability — is a kind ofTraceabilityPRIME

Current abstraction Data Lineage Domain-specific

Parents (1) — more general patterns this builds on

  • Data Lineage is a kind of Traceability Prime

    Data lineage is traceability specialized to data elements and their processing dependency graph.

Hierarchy paths (2) — routes to 2 parentless roots

Neighborhood in Abstraction Space

Data Lineage sits in a sparse region of the domain-specific corpus (85th percentile for distinctiveness): few abstractions share its structure, so a faithful description tends to retrieve it precisely.

Family — Unclustered & Miscellaneous (309 abstractions)

Nearest neighbors

Computed from structural-signature embeddings · 2026-07-12