Data Lineage¶

Recording, for every data element, the complete sequence of sources, transformations, and movements that produced it — as edges in a queryable dependency graph — so audit questions become forward and backward traversals rather than forensic reconstruction.

Version: v4 · 2026-09-28 · History
Domain-specific #: 243
Domain group: Interdisciplinary & Synthetic
Origin domain: Data Science & Analytics
Subdomain: Data Engineering → Data Science & Analytics

Core Idea¶

Data lineage is the discipline of recording, for every data element, the complete sequence of sources, transformations, and movements that produced its current value — with the agents and times at each step — in a form supporting bidirectional traversal of the resulting dependency graph. Backward traversal reaches all upstream sources; forward traversal reaches all downstream consumers. Its distinguishing commitment is queryable graph topology: provenance recorded as graph edges, not a flat audit log.

How would you explain it like I'm…

Cookie Ingredient Cards

Imagine every cookie came with a little card saying which flour, eggs, and sugar it was made from, who mixed it, and when. If a cookie tastes bad, you can follow the cards back to find the problem ingredient, or follow them forward to find every other cookie that used it. Data lineage is those cards for computer information.

Data Family Tree

Data lineage is like a family tree for information in a computer system. For each number or piece of data, it records where it came from, what was done to it, and who or what did it and when. Because all these links are saved as a connected map, you can follow them backward to find the original sources, or forward to see everything that uses that data. So if a number on a chart looks wrong, you can quickly trace back to see where the mistake started. Without lineage, figuring that out can take days of detective work.

Traceable Data Dependency Graph

Data lineage is the practice of recording, for every piece of data in a system, the full chain of sources, transformations, and movements that produced its current value, along with who or what did each step and when. The key idea is that this is stored as a dependency graph, a network of nodes and arrows, not just a flat log. That lets you traverse it in both directions: backward to every upstream source a value depends on, and forward to every downstream report or model it feeds. This turns questions like 'what breaks if I change this source?' or 'why is this dashboard number wrong?' into graph searches. Some regulations require this kind of traceability for important metrics, and tools such as OpenLineage, Apache Atlas, and dbt's lineage views represent these dependencies.

Data lineage is the discipline of recording, for every data element, the complete sequence of sources, transformations, and movements that produced its current value, together with the responsible agents and times, in a form that supports bidirectional traversal of the resulting dependency graph. The substrate is data flowing through a directed acyclic graph, or near-DAG, of processing nodes such as ETL pipelines, joins, aggregations, model training runs, and dashboard computations. What distinguishes lineage from generic documentation or logging is queryable graph topology: provenance is stored as edges, not as a flat audit log. As a result, change-impact analysis becomes forward reachability from a changed source, root-cause analysis becomes backward traversal from a failing output, and blast-radius assessment becomes a reachability computation. Without lineage, the same questions require forensic reconstruction across heterogeneous codebases, which may take days or never finish, whereas with lineage a bad dashboard number can often be resolved quickly. Regulatory frameworks including BCBS 239, GDPR, and FDA data-integrity requirements encode the need for traceable provenance of regulated metrics back to source data. Tooling such as OpenLineage (with Marquez as reference implementation), Apache Atlas, and dbt lineage views represents these dependencies, with OpenLineage integrations mapping other tools' metadata onto one shared graph model.

Scope of Application¶

Lives within data engineering, wherever information flows through a directed (near-DAG) graph of transforming processes whose provenance can be recorded as queryable topology.

Analytics and data warehousing — the home: raw tables through marts to a dashboard tile.
Machine-learning pipelines — which data, code, and hyperparameters produced which model version.
Feature stores — which sources and transformations feed each served feature.
Regulatory compliance — BCBS 239, GDPR, FDA data-integrity traceable provenance.
Scientific reproducibility and cybersecurity forensics — auditing a result's or a compromise's derivation.

Clarity¶

Naming whether a system has lineage makes legible why some failures are tractable and others archaeology — the difference lies not in the team's skill but in whether provenance was recorded as queryable graph topology. It also sharpens what data quality can mean: not a per-snapshot property of a value in isolation, but a property of a value and its derivation, so trust is read off the soundness of the recorded chain.

Manages Complexity¶

Without lineage, every audit question is open-ended forensic work scaling with total code volume. Lineage records provenance once as an explicit graph, reducing the whole family of audit questions to three traversal primitives. The cost signature becomes precise: audit difficulty depends on the depth and fan-out of the graph local to the affected node, not on system size — a bad number resolves as a bounded backward walk in minutes.

Abstract Reasoning¶

Treating the processing DAG as the system model licenses directional moves (forward-reachability for change-impact, backward-traversal for root-cause, reachability for blast-radius), boundary-drawing (tractable-versus-archaeology turns on graph-versus-flat-log; resolution time follows local graph shape), and a further reframe of data quality as a value-and-derivation property, converting trust into a backward-traversal verdict over each node's operation, agent, and refresh time.

Knowledge Transfer¶

Within data engineering the concept transfers literally — the three traversal primitives, the local-cost signature, and the derivation-grounded quality reframe carry across warehousing, ML pipelines, feature stores, and forensics, with tooling (OpenLineage and its reference implementation Marquez, Atlas, dbt) and OpenLineage's per-tool integrations, such as dbt's, mapping lineage onto one graph model. Beyond it the residue is the parents Provenance and Traceability — recurring wherever information moves through transforming processes, but straining once the directed-graph picture goes metaphorical (social influence). The pipeline toolchain and name stay home.

Relationships to Other Abstractions¶

Current abstraction Data Lineage Domain-specific

Parents (1) — outgoing typed relations

Data Lineage is a kind of Traceability Prime

Data lineage is traceability specialized to data elements and their processing dependency graph.

Hierarchy paths (2) — routes to 2 parentless roots

Data Lineage → is a kind of Traceability → presupposes Observability

Show alternative path (1)

Neighborhood in Abstraction Space¶

Data Lineage sits in a sparse region of the domain-specific corpus (78^th percentile for distinctiveness): few abstractions share its structure, so a faithful description tends to retrieve it precisely.

Family — Unclustered & Miscellaneous (2769 abstractions)

Nearest neighbors

Computed from structural-signature embeddings · 2026-10-25