Data Lineage¶

Recording, for every data element, the complete sequence of sources, transformations, and movements that produced it — as edges in a queryable dependency graph — so audit questions become forward and backward traversals rather than forensic reconstruction.

Core Idea¶

Data lineage is the discipline of recording, for every data element in a system, the complete sequence of sources, transformations, and movements that produced its current value — together with the agents and times responsible at each step — in a form that supports bidirectional traversal of the resulting dependency graph. Given any data element, lineage infrastructure allows backward traversal to all upstream sources it transitively depends on and forward traversal to all downstream consumers it transitively feeds. The substrate is information flowing through a directed acyclic graph (or near-DAG) of processing nodes: ETL pipelines, joins, aggregations, model training runs, dashboard computations.

The structural commitment that distinguishes data lineage from generic documentation or logging is queryable graph topology: provenance is not recorded as a flat audit log but as edges in a dependency graph, so change-impact analysis becomes a forward-reachability query from a changed source, root-cause analysis becomes a backward-traversal from a failing output, and blast-radius assessment becomes a reachability computation. Without lineage, the same operations require forensic reconstruction across heterogeneous codebases. The practical payoff is asymmetric: a bad dashboard number with full lineage resolves in minutes through graph traversal; without lineage it is archaeological work that may take days or never complete. Regulatory frameworks including BCBS 239, GDPR, and FDA data-integrity requirements encode this need as mandatory traceable provenance for regulated metrics back to source data. Tooling ecosystems — OpenLineage, Apache Atlas, dbt lineage views, Marquez — have converged on a shared graph model for representing these dependencies across heterogeneous pipeline technologies.

Structural Signature¶

Sig role-phrases:

the data element — the unit whose derivation history is being recorded and reconstructed
the processing graph — the directed acyclic (or near-DAG) graph of nodes — ETL steps, joins, aggregations, training runs, dashboard computations — whose outputs become other nodes' inputs
the transformation record — at each node, the operation performed plus the agent and time responsible
the queryable graph topology — the commitment that provenance is recorded as edges in a dependency graph rather than a flat audit log, so audits become traversals
the backward traversal — from any element, reaching all upstream sources it transitively depends on (root-cause, derivation soundness)
the forward traversal — from any element, reaching all downstream consumers it transitively feeds (change-impact, blast-radius)
the local-cost signature — audit difficulty set by the depth and fan-out of the dependency graph local to the affected node, not by the total code volume of the system
the derivation-grounded quality — trust as a property of a value and its recorded chain of sources, transformations, and refresh times, not of a value in per-snapshot isolation

What It Is Not¶

Not a flat audit log. A chronological record of actions does not encode the transformation graph, so it cannot answer reachability questions. What distinguishes data lineage is queryable graph topology — provenance recorded as edges in a dependency graph — which is exactly what turns change-impact and root-cause into traversals rather than archaeology.
Not generic documentation or logging. Scattered notes and per-job logs describe steps in isolation; lineage is the single recorded derivation graph over which audit questions become forward and backward traversals. The difference is not how much is written down but whether the dependencies are an explicit, traversable structure.
Not versioning. Versioning tracks successive states of the same artifact over time; lineage tracks the derivation across artifacts — which sources and transformations produced this value. One follows an artifact's history; the other follows the graph of what fed what.
Not a per-snapshot data-quality check. Quality-as-validation asks "does this table pass its checks right now?" — a property of a value in isolation. Lineage reframes quality as a property of a value and its derivation: a plausible-looking number can be untrustworthy because its recorded chain includes a stale or unsound upstream step.
Not a measure of system size. With lineage in place, audit difficulty is set by the depth and fan-out of the dependency graph local to the affected node, not by the total code volume — a deep, high-fan-out neighborhood is expensive even in a small system, a shallow one resolves in minutes even in a large one.
Not provenance or traceability as such. Data lineage is the data-engineering specialization of those primes — it adds the substrate restriction that the medium is information flowing through transforming processes (a near-DAG), not artifacts in physical custody or design requirements. The substrate-neutral "trace a derived item to its sources via the derivation graph" belongs to the parents; the ETL/ML-pipeline cargo and the name do not travel.

Scope of Application¶

Data lineage lives within data engineering — wherever information flows through a directed (near-DAG) graph of transforming processes whose provenance can be recorded as queryable graph topology; its reach is within that substrate, since the discipline needs transforming nodes whose edges can be walked. The substrate-neutral residue ("trace a derived item to its sources via the derivation graph") belongs to the parent provenance / traceability primes; the ETL/ML-pipeline toolchain and the name do not travel.

Analytics and data warehousing — the home turf: tracing the path from raw operational tables through staging, fact, and dimension marts to the dashboard tile a stakeholder sees, so a wrong number resolves by backward traversal.
Machine-learning pipelines — recording which training data, code, and hyperparameters produced which model version, for reproducibility and change-impact across model lifecycles.
Feature stores — tracking which upstream sources and transformations feed each served feature, so a changed source's downstream model and serving consumers are enumerable.
Regulatory compliance — BCBS 239, GDPR, and FDA data-integrity requirements encoding mandatory traceable provenance of regulated metrics back to source data.
Scientific reproducibility — recording which raw measurements plus analysis pipeline produced which published figure, so a result's derivation is auditable end to end.
Cybersecurity forensics — reconstructing which input, through which processing nodes, produced an observed corrupted or compromised state.

Clarity¶

Naming whether a system has data lineage makes legible why some failures are tractable and others are archaeology. A wrong dashboard number is, with lineage, a bounded traversal of a recorded dependency graph — walk backward to the nearest upstream node where the value could have gone bad — and without it, a forensic reconstruction stitched across heterogeneous codebases that may take days or never resolve. The label locates the difference precisely: not in the team's skill or the pipeline's complexity, but in whether provenance was recorded as queryable graph topology rather than a flat audit log. That distinction is what separates "we can compute the blast radius of this change" from "we will find out which downstreams broke when they break."

It also sharpens what data quality can even mean. Treated per-snapshot, quality collapses to "does this table pass its validation checks right now?" — a property of a value in isolation. Lineage reframes quality as a property of a value and its derivation: a number is trustworthy not because it looks reasonable but because the chain of sources, transformations, and refresh times that produced it is known and sound. The sharper questions a practitioner can then ask are reachability questions — given this changed source, what transitively depends on it (forward); given this failing output, which upstream node is the closest possible origin (backward) — questions that are well-posed only once provenance is an explicit graph rather than scattered documentation.

Manages Complexity¶

A data system without lineage presents its operators with a black box whose every audit question is an open-ended forensic problem. "Which downstreams break if I change this source?" "Where did this wrong number originate?" "What is the blast radius of this incident?" Each, answered from scratch, means reading across heterogeneous codebases — SQL, ETL scripts, model-training jobs, dashboard definitions — and reconstructing by hand which output fed which input, work whose cost scales with the total code volume of the system and may take days or never resolve. The questions look distinct and each pipeline looks unique, so the team faces a fresh archaeology for every change and every failure. Data lineage collapses that sprawl by recording provenance once as an explicit dependency graph, after which the entire family of audit questions reduces to three traversal primitives on that single object: change-impact is forward-reachability from a changed source, root-cause is backward-traversal from a failing output, and blast-radius is a reachability computation. The heterogeneous codebases stop mattering as such; what the analyst tracks instead is the graph's topology.

The compression has a precise cost signature. Once provenance is queryable graph topology rather than a flat log or scattered documentation, the difficulty of any audit depends on the depth and fan-out of the dependency graph local to the affected node — how many transformations a value passed through, how widely it branches — not on the size of the whole system. A bad dashboard number resolves as a bounded walk backward to the nearest upstream node where the value could have gone wrong, in minutes rather than weeks. And the reframing extends what "data quality" can be tracked as: instead of a per-snapshot property of a value in isolation (does this table pass its checks now?), quality becomes a property of a value together with its derivation, read off the soundness of the recorded chain of sources, transformations, and refresh times. The analyst thus reasons over one well-posed graph and a fixed set of reachability queries, rather than re-deriving each case's history from the raw machinery that produced it.

Abstract Reasoning¶

Data lineage licenses reasoning moves that treat the data-processing DAG as the system model and reduce a family of audit questions to graph traversals on it. The most distinctive is a set of directional moves, each mapping an operational question to one traversal primitive. Change-impact reasoning is a forward move: from a source about to change, the analyst computes forward-reachability to enumerate every downstream consumer that transitively depends on it, predicting the blast radius of the change before it is made rather than discovering which downstreams broke when they break. Root-cause reasoning is the backward move: from a failing output, the analyst walks backward to the nearest upstream node where the value could have gone wrong, narrowing the origin by traversal rather than by reading across heterogeneous codebases. Blast-radius reasoning is a reachability computation over the same graph. The analyst reasons from the direction of the question to the direction of the traversal, and each query is well-posed only because provenance is an explicit graph; the move that makes them answerable is recording provenance as queryable graph topology.

A boundary-drawing move separates the regime where these failures are tractable from the regime where they are archaeology, and locates the difference precisely. The analyst reasons that a wrong dashboard number is, with lineage, a bounded backward traversal of a recorded dependency graph, and without it a forensic reconstruction stitched across SQL, ETL scripts, model-training jobs, and dashboard definitions that may take days or never resolve — and that the difference lies not in the team's skill or the pipeline's complexity but in whether provenance was recorded as a graph rather than a flat audit log. From this the analyst derives a cost prediction: once provenance is queryable topology, the difficulty of any audit depends on the depth and fan-out of the dependency graph local to the affected node — how many transformations a value passed through, how widely it branches — not on the total code volume of the system. So the analyst predicts resolution time from local graph shape, and reasons that a deep, high-fan-out neighborhood is the expensive case even in an otherwise small system, while a shallow one resolves in minutes even in a large one.

A further boundary-drawing move reframes what "data quality" can mean and thereby what can be reasoned about. Treated per-snapshot, quality is a property of a value in isolation — does this table pass its validation checks right now? The analyst instead reasons about quality as a property of a value and its derivation: a number is trustworthy not because it looks reasonable but because the recorded chain of sources, transformations, and refresh times that produced it is known and sound. This converts trust into a backward-traversal verdict — walk the derivation and check each node's operation, agent, and refresh time — and lets the analyst flag a plausible-looking value as untrustworthy when its chain includes a stale or unsound upstream step. The recurring habit the concept installs is to stop asking audit questions of the raw machinery and instead pose them as reachability queries on a single recorded graph: given this changed source, what transitively depends on it; given this failing output, which upstream node is the closest possible origin; and is the derivation behind this value sound end to end.

Knowledge Transfer¶

Within data engineering the concept transfers as mechanism, and the transfer is literal because the substrate is uniform: information flowing through a directed processing graph. The same three traversal primitives (forward-reachability for change-impact, backward-traversal for root-cause, reachability for blast-radius), the same cost signature (resolution time set by the depth and fan-out of the dependency graph local to the affected node, not by total code volume), and the same reframing of data quality (a property of a value and its recorded derivation, not of a value in isolation) carry without translation across analytics and warehousing (raw tables through staging, fact, and dimension marts to a dashboard tile), machine-learning pipelines (which training data, code, and hyperparameters produced which model version), feature stores, scientific-reproducibility pipelines, and cybersecurity forensics. The tooling ports too, because the ecosystems (OpenLineage, Apache Atlas, dbt lineage views, Marquez) deliberately converged on one shared graph model across heterogeneous pipeline technologies, and the regulatory drivers (BCBS 239, GDPR, FDA data-integrity) encode the same traceable-provenance requirement. The vocabulary does not strain across these because they are one substrate: transforming processes connected into a near-DAG of data flow.

Beyond data engineering the honest account is a shared abstract mechanism — case (B), and an unusually clean one because the generalization is exactly the parent the concept specializes. Strip the data-and-pipeline vocabulary and what remains is "trace any derived item back to its sources, and forward to its consumers, via the derivation graph" — which is just provenance (a documented origin-and-custody record) carrying traceability's bidirectional-link commitment. That general pattern genuinely recurs across substrates as co-instances wherever the substrate is information moving through transforming processes — financial-instrument trade lineage, document-change lineage, content-pipeline lineage — and there it carries a real, transferable lesson (record provenance as a queryable graph so audit questions become traversals rather than archaeology). What does not travel is the data-engineering cargo: the ETL/ML-pipeline DAG, the OpenLineage/Atlas/dbt toolchain, the warehouse-specific transformation records, and the "data lineage" name itself are domain furniture that does not survive extraction. And the transfer has a definite edge: it succeeds only while the substrate really is information flowing through transforming processes, and it strains the moment the directed-graph picture goes metaphorical — social influence, political action — where there is no recorded transformation at each node to walk. So the honest move is that the cross-domain reach belongs to the parents (provenance and traceability), of which data lineage is the data-engineering specialization narrowed by a substrate restriction rather than by any new mechanism; "data lineage," as named, does not float free of the data-processing substrate (see Structural Core vs. Domain Accent).

Examples¶

Canonical¶

The dbt (data build tool) lineage graph is the clean worked instance. In dbt, each SQL model declares its inputs with ref() and source() calls, and the tool assembles those declarations into a directed acyclic graph automatically. Consider a small pipeline: stg_orders and stg_currency (sources) → fct_sales (a join/aggregation) → revenue_dashboard (the tile a VP sees). When the dashboard shows an implausible revenue figure, the analyst does not grep across the codebase; they walk the recorded graph backward from revenue_dashboard to its nearest parents, checking each node's last transformation and refresh time, and discover that stg_currency loaded a stale exchange-rate snapshot. Conversely, before editing stg_currency, a forward query over the graph enumerates every model and dashboard that transitively consumes it.

Mapped back: Each dbt model is a node in the processing graph; the ref()/source() declarations are the queryable graph topology — edges, not a flat log. Walking from the bad tile to stg_currency is the backward traversal for root-cause; enumerating consumers before the edit is the forward traversal for change-impact; and flagging the value as untrustworthy because its chain includes a stale refresh is the derivation-grounded quality verdict.

Applied / In Practice¶

Banking regulation deploys lineage at institutional scale through BCBS 239, the Basel Committee's Principles for Effective Risk Data Aggregation and Risk Reporting (issued 2013 for global systemically important banks). After the 2008 crisis, regulators found banks could not reliably trace aggregated risk figures — capital ratios, credit exposures — back to their source records quickly enough to act. BCBS 239 requires banks to maintain traceable provenance so that any reported risk metric can be reconstructed to the underlying transactions and reference data, with documented transformations along the way. Banks implement this as enterprise data-lineage systems over their reporting pipelines, letting auditors and risk officers answer "where did this number come from?" as a backward traversal rather than a months-long reconciliation.

Mapped back: The bank's reporting pipeline is the processing graph; each aggregation and enrichment step logs its operation, owner, and time as the transformation record. Reconstructing a reported capital ratio to its source transactions is the backward traversal, and the mandate that a metric is only trustworthy when its full derivation is documented and sound is the derivation-grounded quality requirement made law.

Structural Tensions¶

T1: Pervasive instrumentation versus its standing overhead (the graph must be paid for before any audit needs it). Recording, for every data element, the operation, agent, and time at each node is not free: every ETL step, join, and training run must emit edges into the dependency graph, and that instrumentation is carried continuously whether or not an audit ever calls on it. The payoff is asymmetric — a bad dashboard number resolves in minutes rather than days — but asymmetric in a way that pays out only when a change, failure, or regulatory question actually arrives. A system that rarely audits pays the standing cost for a rarely-exercised traversal; one under BCBS 239 or FDA mandate cannot afford to be without it. The tension is between paying the recording cost everywhere in advance and capturing provenance only where audits are expected. Diagnostic: Is the expected frequency and blast-radius cost of audits high enough to justify emitting provenance at every node rather than only the hot paths?

T2: Local-cost promise versus deep, high-fan-out neighborhoods (the same signature that reassures also warns). The headline reassurance is that, once provenance is queryable topology, audit difficulty depends on the depth and fan-out of the graph local to the affected node, not on total code volume — so a wrong number in a large system can resolve in minutes. But that same signature is a warning: a deep, high-fan-out neighborhood is expensive to audit even inside an otherwise small system, and no amount of lineage tooling flattens it. Lineage relocates the cost from "size of the whole system" to "shape of the local graph," which helps only where that shape is benign. Diagnostic: Is the affected node's local neighborhood shallow and narrow, or does the value pass through many deeply-nested, widely-branching transformations?

T3: Recorded graph versus the real pipeline (traversal trusts what was captured, not what ran). Every traversal — forward for change-impact, backward for root-cause — is only as sound as the graph's correspondence to the transformations that actually executed. A manual SQL fix, an undocumented enrichment, or a step that mutates data outside the recorded nodes creates a value whose true derivation the graph does not contain, and the traversal then returns a confident but false answer, indistinguishable from a correct one. The very move that makes audits tractable — trusting the graph instead of re-reading heterogeneous codebases — is what makes an incomplete graph dangerous, because it retires the forensic habit that would have caught the gap. Diagnostic: Is every transformation guaranteed to flow through a recorded node, or can data reach an output by a path the graph never saw?

T4: Bidirectional traversal versus feedback (the DAG picture that licenses reachability can be violated by the substrate). The whole apparatus presupposes a directed acyclic (or near-DAG) graph: "all upstream sources" and "all downstream consumers" are well-posed only because the dependency graph does not loop. But real data systems contain feedback — a model whose predictions are logged and later become training data, a metric that drives an action that changes a source — and where such an edge exists, "upstream" and "downstream" stop being cleanly separable, and reachability can enumerate a value as both its own ancestor and its descendant. The clean forward/backward split is a property of the near-DAG assumption, not a guarantee the substrate honors. Diagnostic: Does the pipeline contain feedback edges that make the upstream/downstream distinction ill-defined for the value under audit?

T5: Autonomy versus reduction (its own tooled discipline or the data-engineering instance of its parents). "Data lineage" is a named, tooled discipline — OpenLineage, Atlas, dbt lineage views, Marquez, the BCBS 239 / GDPR / FDA mandates — with its own ETL/ML-pipeline cargo. Strip that cargo and what remains is provenance (a documented origin-and-custody record) carrying traceability's bidirectional-link commitment: trace any derived item to its sources and its consumers via the derivation graph. That residue is exactly what recurs in trade lineage, document lineage, and content pipelines, and it is the parents, not the term, that carry it. The tension is between a standalone data-engineering label that earns its own toolchain and study and the recognition that its cross-domain reach belongs to provenance and traceability. Diagnostic: Resolve toward the parents when asking what the lesson is outside data pipelines; toward "data lineage" when diagnosing a specific value's derivation inside a processing DAG.

Structural–Framed Character¶

Data lineage sits at the mixed point of the spectrum — a neutral recording discipline laid over a genuinely real computational structure, so it straddles: framed in its documentation-and-tooling apparatus, structural in the derivation graph it records.

On evaluative weight it is neutral: recording provenance and walking the derivation graph convict nothing — "data lineage" praises and blames no value, it only makes a value's derivation legible. Like feedback or isostasy, the discipline itself renders no verdict. On human_practice_bound it is genuinely mixed: the concept is the discipline of recording provenance as queryable topology — a human practice, and T3 is explicit that the recorded graph can diverge from the pipeline that actually ran — yet the thing recorded is a real fact about the system, since the data truly did flow through those transformations whether or not anyone logged them. So it is neither observer-free physics nor a pure practice-artifact, but a documentation practice over an observer-independent dependency structure. Its institutional_origin is likewise split: the derivation graph is a fact about the running pipeline, while the tooling (OpenLineage, Atlas, dbt lineage, Marquez) and the mandates (BCBS 239, GDPR, FDA data-integrity) that make "data lineage" a named discipline are institutional artifacts. On vocab_travels it scores low: ETL, near-DAG, warehouse marts, the specific toolchain, and the "lineage" name are pinned to the data-engineering substrate. And import_vs_recognize is bimodal in the entry's own terms — within data engineering the mechanism is recognized intact (analytics, ML pipelines, feature stores, forensics), while beyond it the parents carry trade/document/content lineage as co-instances and the picture strains wherever the derivation graph goes metaphorical.

The one portable structural skeleton is a bidirectionally-traversable derivation graph — record, for every derived item, the sources and transformations that produced it as edges in a queryable dependency graph, so audit becomes backward traversal to origins and forward traversal to consumers. This decomposes cleanly into provenance (a documented origin-and-custody record) carrying traceability's bidirectional-link commitment. That skeleton genuinely travels and recurs as co-instances wherever information moves through transforming processes, which tempts a structural reading. But it does not lift data lineage off the mixed point, because the entry is explicit that it is the data-engineering specialization of exactly those parents — narrowed by a substrate restriction, not by any new mechanism — so the traversable-derivation-graph structure is what data lineage instantiates from provenance/traceability, not what makes "data lineage" itself travel: the cross-domain reach belongs to the parents, while the ETL/ML-pipeline DAG, the tooling ecosystem, the warehouse-specific transformation records, and the name — the distinctive cargo — stay home. Its character: an evaluatively neutral recording discipline over a real computational dependency structure, structural in the traversable-derivation-graph skeleton it borrows from provenance and traceability, and framed in the data-engineering tooling, mandates, and vocabulary that are its specialization rather than its portable core.

Structural Core vs. Domain Accent¶

This section decides why data lineage is a domain-specific abstraction and not a prime, and it carries the case for its domain-specificity — there is no separate section for that.

What is skeletal (could lift toward a cross-domain prime). Strip the data engineering away and a thin relational structure survives: record, for every derived item, the sources and transformations that produced it as edges in a queryable dependency graph, so audit becomes backward traversal to origins and forward traversal to consumers. The portable pieces are abstract — a derivation graph rather than a flat log, three traversal primitives (forward-reachability for change-impact, backward-traversal for root-cause, reachability for blast-radius), a cost that depends on the local graph's depth and fan-out rather than the whole system's size, and trust reframed as a property of a value and its recorded chain rather than a value in isolation. That traversable-derivation-graph skeleton is genuinely substrate-portable: it recurs, as the entry stresses, as co-instances wherever information moves through transforming processes — financial-instrument trade lineage, document-change lineage, content-pipeline lineage. That recurrence is mechanism, which is why the entry calls data lineage the specialization of exactly its parents — but it is the core data lineage shares, not what makes it data lineage.

What is domain-bound. Almost everything that makes the concept data lineage in particular is data-engineering furniture and none of it survives extraction intact: the processing DAG of ETL steps, joins, aggregations, training runs, and dashboard computations; the tooling ecosystem (OpenLineage, Apache Atlas, dbt lineage views, Marquez) that converged on one shared graph model; the warehouse-specific transformation records; the regulatory mandates (BCBS 239, GDPR, FDA data-integrity) that encode traceable provenance as law; and the empirical cases — the dbt ref()/source() graph resolving a stale exchange rate, the bank reconstructing a capital ratio to its source transactions. These are the worked vocabulary, toolchain, and cases the discipline actually studies. The decisive test: the concept narrows its parents by a substrate restriction — the medium must be information flowing through transforming processes whose edges can be walked — and it strains the moment the directed-graph picture goes metaphorical (social influence, political action), where there is no recorded transformation at each node to traverse; remove the data-processing substrate and what remains is bare "trace a derived item to its sources," a looser and more general thing.

Why this does not clear the prime bar. A prime is a relational structure whose vocabulary travels and whose cross-domain transfer is recognition of the same mechanism, not analogy. Data lineage's transfer is bimodal. Within data engineering the mechanism travels intact — the three traversal primitives, the local-cost signature, and the derivation-grounded reframing of data quality all keep their meaning across analytics and warehousing, ML pipelines, feature stores, scientific-reproducibility pipelines, and cybersecurity forensics, because all are one substrate: transforming processes connected into a near-DAG of data flow, with a toolchain that deliberately ports across them. Beyond data engineering, the transfer succeeds only while the substrate really is information flowing through transforming processes, and it strains into analogy the moment the graph goes metaphorical. And when the bare structural lesson is needed cross-domain, it is already carried, in more general form, by the parents data lineage specializes: a documented origin-and-custody record is provenance, and the bidirectional-link commitment (trace to sources and to consumers) is traceability — of which data lineage is the data-engineering specialization narrowed by a substrate restriction rather than by any new mechanism. The cross-domain reach belongs to those parents; "data lineage," as named, does not float free of the data-processing substrate, carrying ETL/ML-pipeline, tooling, and mandate baggage that does not and should not travel.

Relationships to Other Abstractions¶

Current abstraction Data Lineage Domain-specific

Parents (1) — more general patterns this builds on

Data Lineage is a kind of Traceability Prime

Data lineage is traceability specialized to data elements and their processing dependency graph.

Hierarchy paths (2) — routes to 2 parentless roots

Data Lineage → Traceability → Observability

Show alternative path (1)

Not to Be Confused With¶

Data versioning / data version control. Tracks successive states of the same artifact over time — snapshots, diffs, rollback (Git, DVC, table time-travel). Lineage tracks the derivation across artifacts: which sources and transformations produced this value. One follows an artifact's own history through revisions; the other follows the graph of what fed what. Tell: is the question "what did this table look like last week?" (versioning) or "which sources and steps produced this value?" (lineage)? They compose but answer different axes.
Data catalog / metadata management. The broader discovery-and-inventory system describing what datasets exist, their schemas, owners, and descriptions. Lineage is one capability often surfaced within a catalog — the dependency-graph layer — not the catalog itself. Part-vs-whole: the catalog is the registry of assets; lineage is the recorded edges between them. Tell: is the concern what data exists and how it's described (catalog) or how each value was derived from others (lineage)?
Pipeline observability / monitoring. Runtime operational telemetry — job success/failure, latency, throughput, freshness alerts — that tells you whether the pipeline is healthy now. Lineage is the structural record of derivation, telling you what depends on what regardless of current run state. Tell: does the signal report the operational health of executions (observability) or the dependency topology for tracing impact and origin (lineage)? Monitoring watches the runs; lineage maps the graph.
Data governance. The organizational discipline of policies, ownership, access control, and compliance over data. Lineage is an enabling substrate for governance (it makes traceability mandates like BCBS 239 satisfiable) but is not itself the policy layer. Tell: is the referent the rules and accountability over data (governance) or the recorded derivation graph those rules rely on (lineage)?
The provenance / traceability parents (umbrella). The substrate-neutral primes data lineage specializes — provenance (a documented origin-and-custody record) carrying traceability's bidirectional-link commitment. Not confusable peers but the generalization: trade lineage, document lineage, and physical chain-of-custody are co-instances of the parents. Data lineage narrows them by a substrate restriction (information flowing through transforming near-DAG nodes) rather than any new mechanism. Tell: the parents travel to any derived item traceable to its sources; "data lineage," treated more fully in a later section, is the data-engineering specialization with its ETL/ML DAG and tooling.

Neighborhood in Abstraction Space¶

Data Lineage sits in a sparse region of the domain-specific corpus (85^th percentile for distinctiveness): few abstractions share its structure, so a faithful description tends to retrieve it precisely.

Family — Unclustered & Miscellaneous (309 abstractions)

Nearest neighbors

Computed from structural-signature embeddings · 2026-07-12