What Is Data Lineage? Definition, Examples, and How It Works

What is data lineage? Data lineage is the documented path data takes from its original sources to its final destinations, including every table, view, job, and transformation it passes through along the way. It answers two questions every data team asks: “Where did this value come from?” (provenance) and “What depends on it downstream?” (impact analysis). In SQL-based systems, lineage is usually extracted automatically by parsing the SQL code that moves and reshapes the data.

This page is a working definition for practitioners: what lineage actually records, the difference between table-level and column-level granularity, direct versus indirect relationships, the three ways lineage gets built, and who genuinely needs it. Examples use SQL, because in most warehouses SQL is where the transformation logic lives.

The two questions lineage answers: provenance and impact analysis

Data lineage is one graph read in two directions. Read it upstream and you get provenance: starting from a figure in a report, walk backward through every view, join, and calculation to the raw source columns it was derived from. This is the direction auditors, regulators, and anyone debugging a wrong number care about.

Read it downstream and you get impact analysis: starting from a column you want to rename, retype, or drop, enumerate every table, view, procedure, and report that would be affected. This is the direction engineers care about before a schema change or a migration, when the alternative is grepping thousands of scripts and hoping.

The terms are sometimes used loosely. Strictly, provenance is the backward trace, impact analysis is the forward trace, and data lineage is the underlying dependency graph that makes both traversals possible.

A concrete example

Consider a routine warehouse statement:

INSERT INTO monthly_revenue (month, total)
SELECT DATE_TRUNC('month', o.order_date) AS month,
       SUM(o.amount)                     AS total
FROM   orders o
JOIN   customers c ON o.customer_id = c.id
WHERE  c.region = 'EMEA'
GROUP  BY DATE_TRUNC('month', o.order_date);

The lineage extracted from this one statement already has two distinct kinds of relationship:

• Direct lineage — data actually flows into the output: orders.order_date feeds monthly_revenue.month through DATE_TRUNC, and orders.amount feeds monthly_revenue.total through SUM.
• Indirect lineage — columns that never land in the output but shape it: customers.region filters the rows, and orders.customer_id joined to customers.id decides which rows match at all. Change how region is populated and every number in monthly_revenue changes, even though region appears nowhere in the result.

A real pipeline is thousands of statements like this, layered through views, stored procedures, and dbt models. Lineage tooling exists because no human keeps that graph in their head. For more worked cases, including multi-hop and stored-procedure lineage, see our data lineage examples walkthrough.

Table-level vs column-level lineage

Lineage comes in two granularities, and the difference determines what you can actually do with it.

	Table-level lineage	Column-level lineage
What it records	monthly_revenue is built from orders and customers	monthly_revenue.total = SUM(orders.amount), filtered by customers.region, joined on customer_id = id
Impact analysis	Approximate: flags every consumer of the table, including ones the changed column never touches	Precise: flags only the consumers the column actually feeds
Audit answers	“This report uses these tables”	“This figure is computed from exactly these source fields, via these transformations”
Typical source	Job orchestrator metadata, query logs	Parsing the SQL itself

Table-level lineage is cheap to collect and useful for a first map of a pipeline. But most real questions are column questions: which reports use email, which source fields feed this regulated figure, is anything downstream reading the column we want to drop. That requires resolving each output column through joins, subqueries, CTEs, views, and SELECT * expansion: parser work, not log scraping. The mechanics are covered in depth in our guide to column-level data lineage.

How data lineage is built: three approaches

Every lineage tool on the market uses some mix of three techniques.

Approach	How it works	Strong at	Weak at
Static SQL analysis	Parse the SQL code (views, procedures, ETL scripts, dbt models) and derive the dependency graph from the syntax and semantics	Column-level precision; covers all code paths, including ones that haven’t run recently; needs no access to data	Needs a parser that genuinely understands each SQL dialect; non-SQL transformations (Python, Spark DataFrames) are out of scope
Runtime / query-log based	Observe what actually executed — warehouse query history, orchestrator events (the OpenLineage/Marquez model)	Capturing real executions across heterogeneous jobs, including non-SQL ones; ground truth for “what ran when”	Only sees paths that executed during the observation window; granularity is often table-level; logs still contain SQL that must be parsed for column detail
Manual documentation	Analysts record mappings in spreadsheets or a catalog by hand	Business context and definitions no parser can infer	Stale the day after it’s written; error-prone at any real scale

These are complements, not rivals: runtime lineage tells you what ran, manual curation adds business meaning, and static SQL analysis supplies the column-level logic map. Since most warehouse transformation logic is SQL, static analysis does the heaviest lifting, and its quality is exactly the quality of its parser. Gudu SQLFlow takes this approach with 39 dialect-specific parsers (Snowflake, BigQuery, Oracle, SQL Server, Teradata, and 34 more), built on a SQL compiler front-end developed since the mid-2000s and validated against roughly 13,600 per-dialect test fixtures. That depth is what lets it handle the SQL that breaks simpler parsers: Oracle PL/SQL and T-SQL stored procedures, dynamic SQL assembled at runtime, temp tables, and view stacks.

What is data lineage used for?

Four use cases account for nearly all lineage adoption in practice:

• Regulatory compliance. BCBS 239 expects banks to demonstrate how risk figures are aggregated from source data; GDPR data-mapping obligations require knowing which systems and derived tables personal data flows into. Column-level provenance is the artifact that satisfies both: “these exact source fields, through these transformations.”
• Data governance. Catalogs and glossaries go stale when lineage is maintained by hand. Automatically extracted lineage keeps the dependency map current as code changes, and can feed catalog platforms directly; SQLFlow, for example, ships export adapters for DataHub, Microsoft Purview, and OpenMetadata.
• Debugging and root-cause analysis. When a dashboard shows a wrong number, lineage turns “search every script that mentions this table” into “walk backward from this one column.” Hours instead of days.
• Migration and modernization. Before moving from Oracle or Teradata to Snowflake, BigQuery, or Databricks, the dependency graph tells you what must move together, in what order, and which objects nothing reads anymore and can be dropped rather than migrated.

A useful heuristic: if your team has ever delayed a schema change because nobody was sure what it would break, you already need lineage — you’re just computing it manually, slowly, and with errors.

Frequently asked questions

What is data lineage in simple terms?

It is the map of where your data comes from and where it goes: which sources feed each table or report, and what transformations happen in between. Think of it as a dependency graph for data, traversable backward (provenance) and forward (impact analysis).

What is the difference between data lineage and data provenance?

Provenance is the backward trace from one output to its origins. Lineage is the whole dependency graph, which supports provenance in one direction and impact analysis in the other. In everyday usage the terms overlap, but the graph-versus-traversal distinction is the precise one.

What is indirect data lineage?

Indirect lineage covers columns that influence a result without appearing in it — columns used in WHERE filters, JOIN conditions, and GROUP BY clauses. They change every output value, so impact analysis that ignores them is incomplete. SQLFlow models indirect lineage as a separate, toggleable relationship type; most lineage tools do not make the distinction.

Is data lineage required for GDPR or BCBS 239 compliance?

Neither framework names “lineage” as a mandatory technology, but both require what lineage provides: BCBS 239 asks banks to show how risk data is aggregated from its sources, and GDPR data mapping requires knowing where personal data flows. Column-level lineage is the standard way to produce that evidence at scale.

How do you create data lineage automatically?

Feed your SQL to a static-analysis lineage tool. With SQLFlow you can paste SQL, upload files, connect a database over JDBC, import a dbt manifest, or ingest Snowflake query history or Redshift query logs; it parses the code and produces interactive column-level diagrams plus JSON/CSV exports and a REST API. Enterprise deployments batch-scan estates of 100+ databases and over a million columns.

Does building lineage require access to my actual data?

Not with static analysis. Parsing SQL requires the code and, optionally, schema metadata (table and column definitions) — never the rows in your tables. SQLFlow’s On-Premise edition additionally keeps the SQL text itself inside your network, which is why it fits banks and other regulated environments.