Data Lineage Record¶
Provenance record — instantiates Traceability Linking
Follows a data value back through every source, join, and transformation that produced it and forward to everything that now depends on it, pinning each step to the pipeline version and the steward who owns it.
A number on a dashboard is the end of a long chain nobody can see. A Data Lineage Record makes that chain explicit for data specifically: it traces a dataset, table, or column back through every source, join, filter, and transformation that produced it, and forward to every report, model, or feed that consumes it. Its distinguishing detail is that each edge is pinned to the pipeline version that produced it and owned by a data steward who keeps it accurate — so a value can be tied to the exact transformation logic that made it, at the time it ran. It is not a judgement of whether the source data is any good (that is provenance evaluation) nor a ruling on which store is authoritative (source-of-truth assignment); it maps the path the data actually took.
Example¶
A finance dashboard shows monthly recurring revenue trending down, and nobody believes it. Without lineage, confirming the figure means auditing the whole stack by hand. With a Data Lineage Record, an analyst walks the graph backward: the MRR tile ← a summary model ← a revenue table ← a join of two source systems — one of which changed how it handles multi-currency contracts in last week's release.
Column-level lineage[n1] pins the problem to a single upstream field rather than a whole pipeline, and the version anchor shows the change landed exactly when the dip began. The fix is surgical. Run the same graph forward from that field and it also answers the other question teams dread — "what else breaks if we correct this?" — by listing every downstream table and report that draws on it.
How it works¶
It captures origin → transformation → destination at the granularity the situation needs — dataset, table, or ideally column. Three things make it more than a diagram: each transformation edge is version-pinned to the pipeline or code revision that produced it, so a value ties to the exact logic and moment of its creation; each segment has an owner — a steward or catalog team — because lineage that nobody maintains decays the instant a pipeline changes; and edges are ideally auto-extracted from query and pipeline metadata rather than hand-drawn, so the graph tracks reality instead of documenting an intention. The result is followable both ways: back to origin, forward to consumers.
Tuning parameters¶
- Granularity — table-level versus column-level versus value-level lineage. Column-level pinpoints the offending field but is far more expensive to capture and store.
- Capture method — auto-parsed from pipeline/query metadata versus hand-maintained. Automated capture stays current; manual capture drifts but can record logic tools can't see.
- Version-pinning depth — whether each edge records the exact code/pipeline revision or just "current." Deeper pinning ties a value to the logic that made it; shallow pinning loses the past.
- Ownership model — a per-dataset steward versus a central catalog team. Distributed ownership scales with the org; central ownership gives consistency but a bottleneck.
- Downstream reach — how far forward consumption is tracked (immediate reports vs. the full transitive fan-out).
When it helps, and when it misleads¶
Its strength is turning two dreaded questions — "why is this number wrong?" and "what breaks if I change this source?" — into a graph query instead of a manual audit, and scoping a change's reach before it ships.
Its failure modes cluster around staleness and blind spots. Lineage drifts out of date the moment a pipeline changes without re-capture, and stale lineage is worse than none because it inspires confidence in a map that no longer matches the territory. Auto-extracted lineage misses logic hidden in application code, notebooks, or manual spreadsheet steps, so the graph can look complete while a real edge is invisible. And like an audit trail, it shows the path a value took, not whether the value is correct or the source trustworthy — a clean lineage graph over bad data is still bad data with good directions. The discipline that keeps it honest is to prefer auto-capture bound to the pipeline, give every segment an owner, and treat lineage as needing refresh on every change rather than as a one-time diagram.
How it implements the components¶
traceable_unit— datasets, tables, and columns are the identifiable units the lineage graph links and reviewers inspect.version_anchor— each transformation edge is pinned to the pipeline/code version and time it was valid, tying a value to the exact logic that produced it.link_owner— a data steward owns each lineage segment; without an owner the graph decays as pipelines evolve.
It does not summarize the downstream blast radius of a change as a report (impact_link — Change Impact Report), type the support relation between claims (link_semantics — Citation Chain), or monitor coverage and stale-link exceptions across the system (Traceability Dashboard).
Related¶
- Instantiates: Traceability Linking — the Data Lineage Record supplies the data-origin-to-consumption strand of the trace network.
- Sibling mechanisms: Audit Trail Record · Change Impact Report · Citation Chain · Decision Log · Requirements Traceability Matrix · Chain-of-Custody Record · Source Control Linkage · Test Coverage Link · Traceability Dashboard
Editorial Notes¶
Form Classification¶
Form family: Record, Log & Register
Rationale: Data Lineage Record operates as a durable record, ledger, register, or trace whose value depends on preserving actual state or history because it follows a data value back through every source, join, and transformation that produced it and forward to everything that now depends on it, pinning each step to the pipeline version and the steward who owns it.
Independent corroboration: The frozen evidence defines Data Lineage Record as 'Follows a data value back through every source, join, and transformation that produced it and forward to everything that now depends on it, pinning each step to the pipeline version and the steward who owns it', so its operative form is Record, Log & Register.
Review outcome: Independent reviewer agreement; high confidence.
Origin Attribution¶
Primary origin: Data Science & Analytics
Origin pattern: Cross-disciplinary synthesis
Present-day reach: Specialized
Rationale: Data governance cohered lineage records tracing datasets and columns through sources, joins, filters, transformations, pipeline versions, stewards, and downstream consumers.
Related originating lineages:
- Computer Science & Software Engineering — Database and workflow systems supplied executable dependency graphs and version-pinned transformation metadata.
- Library & Information Science — Provenance and metadata stewardship supplied durable descriptive records and accountable custodianship.
Review resolution: The record combines data-engineering dependency provenance with database metadata and documentary catalog traditions, forming a specialized cross-disciplinary synthesis.
Review outcome: Reconciled after independent review; high confidence.
Notes¶
A Data Lineage Record shows the path a value took; it does not certify the value is accurate or the source authoritative — those are Data Integrity Preservation and Source-of-Truth Assignment, separate archetypes. The most dangerous state is silent staleness: a lineage graph that has stopped tracking a changed pipeline gives precise, confident, and wrong answers.
[n1] Column-level lineage traces provenance at the granularity of individual fields rather than whole tables, letting a wrong value be pinned to the exact upstream column and transformation that produced it; open standards such as OpenLineage exist to capture it automatically from pipeline metadata. ↩