Databricks Data Lineage: Column-Level Lineage for Spark SQL and Delta

Databricks data lineage is the column-level map of how data flows through your Spark SQL and Delta Lake code: which source columns feed each target table across your bronze, silver, and gold layers, and which transformations — joins, aggregates, MERGE logic — happen along the way. Unity Catalog captures lineage automatically for workloads that run through it; Gudu SQLFlow covers everything else by parsing the SQL itself, which makes it the tool for migrations into Databricks, SQL that runs outside Unity Catalog, and estates that span Databricks plus other platforms.

When do you need more than Unity Catalog lineage?

Unity Catalog is genuinely good at what it does: for queries and jobs executed through it, Databricks captures lineage automatically, with no extra tooling. If your whole estate lives inside one Databricks account and every workload runs through Unity Catalog, start there.

The gap is structural, not a bug: runtime-captured lineage only exists for code that has already executed in the environment doing the capturing. That leaves four situations where teams reach for a SQL-parsing approach instead:

• Before the code runs on Databricks. You are migrating from Oracle, Teradata, or SQL Server and need the dependency graph of the legacy SQL — and of the rewritten Databricks SQL — before anything executes in production.
• SQL that never touches Unity Catalog. Notebook SQL exported to files, scripts in a Git repo, SQL generated by external schedulers or ETL tools, code on workspaces not yet upgraded to Unity Catalog.
• Cross-platform estates. Pipelines that start in SQL Server, transform in Databricks, and land in Snowflake need one lineage graph across all three. SQLFlow ships dialect-specific parsers for 39 databases — Databricks and Spark SQL among them — so the whole chain is analyzed with one engine.
• Lineage in a catalog you already run. If your organization standardizes on DataHub, Microsoft Purview, or OpenMetadata rather than the Databricks UI, you need lineage that exports there. SQLFlow includes export adapters for all three.

The two approaches are complementary. Unity Catalog tells you what ran; static SQL analysis tells you what the code does — including code that hasn’t run yet.

How SQLFlow parses Databricks SQL

Gudu SQLFlow is an automated SQL data lineage tool built on the General SQL Parser, a commercial SQL compiler front-end developed since the mid-2000s and validated against roughly 13,600 per-dialect test fixtures. It ships a dedicated Databricks dialect parser in the Spark SQL family — not a generic ANSI grammar — so Databricks-specific constructs are parsed as first-class syntax rather than approximated.

The analysis is entirely static. SQLFlow reads SQL text and, optionally, schema metadata; it never reads the rows in your Delta tables, and with the On-Premise edition the SQL text itself never leaves your network: it deploys on Docker or Kubernetes and runs air-gapped. For each statement, the parser builds a full semantic model, resolving every column reference through CTEs, subqueries, views, and SELECT * expansion, then the data-flow analyzer extracts source-to-target relationships at column granularity.

You can feed it SQL any way it exists in your estate: pasted statements, uploaded files, database metadata over JDBC, or a dbt manifest for dbt projects targeting Databricks. Output is an interactive, drillable diagram plus structured lineage data as JSON, CSV, PNG, or a REST API response.

Example: column-level lineage through a Delta MERGE

MERGE INTO is the workhorse of Delta Lake pipelines, and a statement where table-level lineage is nearly useless, because a single merge reads, matches, updates, and inserts in one shot. Consider a gold-layer upsert:

MERGE INTO gold.customer_ltv AS t
USING (
  SELECT o.customer_id,
         SUM(o.amount)   AS lifetime_value,
         MAX(o.order_ts) AS last_order_ts
  FROM   silver.orders o
  WHERE  o.status = 'completed'
  GROUP BY o.customer_id
) AS s
ON t.customer_id = s.customer_id
WHEN MATCHED THEN UPDATE SET
  t.lifetime_value = s.lifetime_value,
  t.last_order_ts  = s.last_order_ts
WHEN NOT MATCHED THEN INSERT
  (customer_id, lifetime_value, last_order_ts)
  VALUES (s.customer_id, s.lifetime_value, s.last_order_ts);

SQLFlow resolves this into precise column-level relationships. gold.customer_ltv.lifetime_value derives from silver.orders.amount through SUM(), via both the UPDATE and the INSERT branch. last_order_ts derives from silver.orders.order_ts through MAX(). The subquery alias s is resolved away; lineage points at the real source table, not the intermediate.

Just as important is what SQLFlow classifies as indirect lineage: silver.orders.status never lands in the target, but the WHERE filter on it shapes every merged value, and customer_id drives both the GROUP BY and the match condition. SQLFlow models direct dataflow and indirect influence as distinct, separately toggleable relationship types. Most competing tools do not make that distinction, and it is exactly what you need when someone asks “will changing the status vocabulary break customer LTV?” The honest answer is yes, and only impact-aware lineage shows it.

Migrating to (or from) Databricks

Migration is one of the most common reasons teams analyze Databricks SQL outside the platform. Unity Catalog cannot help you plan a migration, because the lineage you need describes code that has never run on Databricks. A parser-based workflow does:

1. Map the source estate. Run the legacy SQL — Oracle, Teradata, SQL Server, including stored procedures and the dynamic SQL inside them — through SQLFlow to get the true dependency graph. This tells you what to migrate first and what is dead code.
2. Validate the rewrite. Analyze the rewritten Databricks SQL with the Databricks dialect parser and compare lineage graphs. If a target column’s sources changed between the Teradata version and the Delta version, you have found a rewrite bug before it ships.
3. Verify nothing is orphaned. After cutover, batch-scan both estates — SQLFlow handles estates of 100+ databases and over a million columns, with incremental scans into a persistent lineage repository.

The same cross-dialect capability works in reverse or sideways: if part of your warehouse is moving to Snowflake instead, the Snowflake lineage workflow uses the identical engine, so the two platforms end up in one comparable graph.

Exporting Databricks data lineage to DataHub, Purview, or OpenMetadata

Catalog-first platforms are good systems of record for metadata, and many organizations mandate one across all data platforms. SQLFlow slots in as the lineage engine underneath: it parses the SQL, computes column-level and indirect lineage, and pushes the result through dedicated export adapters for DataHub, Microsoft Purview, and OpenMetadata. JSON and CSV exports plus a REST API cover anything custom.

This is also the practical answer to the multi-platform problem: lineage produced by two different extractors with two different models is hard to stitch into one graph. One parser producing one graph for all 39 dialects, exported to the catalog you already run, avoids the problem.

Ways to run it

Option	Best for	Notes
SQLFlow Cloud	Trying it on real Databricks SQL today	Free tier in the browser; premium $49.99/month
SQLFlow On-Premise	Regulated environments; SQL must stay internal	Docker/Kubernetes, air-gapped capable; $500/month or $4,800 one-time per database type, installable on two servers
REST API / CLI / Java library	Automating lineage in CI or your own platform	Same engine, headless; embeddable JavaScript widget with a 30+ method API for rendering diagrams in your product

Frequently asked questions

Does SQLFlow replace Unity Catalog lineage?

No, it complements it. Unity Catalog captures lineage for workloads that run through it, automatically. SQLFlow analyzes the SQL statically, so it covers code before it runs (migrations), SQL that executes outside Unity Catalog, and estates spanning Databricks plus other platforms, then exports the result to DataHub, Purview, or OpenMetadata.

Does SQLFlow need access to my Databricks workspace or data?

No. SQLFlow performs static analysis of SQL code, optionally using schema metadata to resolve references. It never reads rows in your Delta tables. With the On-Premise edition, even the SQL text stays inside your network.

Can SQLFlow trace lineage through MERGE INTO statements?

Yes. For a Delta MERGE INTO, SQLFlow resolves the USING subquery, maps each target column to its true source columns through both the UPDATE and INSERT branches, and classifies match conditions and filters as indirect (impact) lineage.

Is Databricks a distinct dialect from Spark SQL in SQLFlow?

Yes. SQLFlow’s 39 supported dialects list Databricks and Spark SQL separately, each with its own parser in the Spark SQL family. Pick Databricks for SQL written against Databricks; there is also a dedicated Spark SQL lineage page for open-source Spark workloads.

Does it work with dbt projects on Databricks?

Yes. Import the dbt manifest and SQLFlow produces column-level lineage across your models, using the Databricks dialect to parse the compiled SQL.

What does SQLFlow cost?

SQLFlow Cloud starts free; premium is $49.99/month. On-Premise is $500/month or $4,800 one-time per selected database type. Full details are on the pricing page.