{"id":6753,"date":"2026-07-12T02:06:31","date_gmt":"2026-07-12T10:06:31","guid":{"rendered":"https:\/\/www.gudusoft.com\/databricks-data-lineage\/"},"modified":"2026-07-12T02:06:31","modified_gmt":"2026-07-12T10:06:31","slug":"databricks-data-lineage","status":"publish","type":"page","link":"https:\/\/www.gudusoft.com\/pt\/databricks-data-lineage\/","title":{"rendered":"Databricks Data Lineage: Column-Level Lineage for Spark SQL and Delta"},"content":{"rendered":"<p><strong>Databricks data lineage<\/strong> is the column-level map of how data flows through your Spark SQL and Delta Lake code: which source columns feed each target table across your bronze, silver, and gold layers, and which transformations \u2014 joins, aggregates, <code>MERGE<\/code> logic \u2014 happen along the way. Unity Catalog captures lineage automatically for workloads that run through it; <strong>Gudu SQLFlow<\/strong> covers everything else by parsing the SQL itself, which makes it the tool for migrations into Databricks, SQL that runs outside Unity Catalog, and estates that span Databricks plus other platforms.<\/p>\n\n\n\n<div class=\"wp-block-group alignfull has-background is-layout-constrained\" style=\"background-color:#eef7fb;padding-top:24px;padding-bottom:24px\"><div class=\"wp-block-group__inner-container\">\n\n<p><strong>Try it in 30 seconds:<\/strong> paste a Databricks <code>MERGE INTO<\/code> or notebook query into the <a href=\"https:\/\/sqlflow.gudusoft.com\/?utm_source=gudusoft&amp;utm_medium=website&amp;utm_campaign=databricks-data-lineage\" target=\"_blank\" rel=\"noreferrer noopener\">free SQLFlow lineage visualizer<\/a>, select the Databricks dialect, and get the column-level lineage diagram immediately. No cluster and no workspace access required.<\/p>\n\n<\/div><\/div>\n\n\n\n<h2 class=\"wp-block-heading\">When do you need more than Unity Catalog lineage?<\/h2>\n\n\n\n<p>Unity Catalog is genuinely good at what it does: for queries and jobs executed through it, Databricks captures lineage automatically, with no extra tooling. If your whole estate lives inside one Databricks account and every workload runs through Unity Catalog, start there.<\/p>\n\n\n\n<p>The gap is structural, not a bug: runtime-captured lineage only exists for code that has already executed in the environment doing the capturing. That leaves four situations where teams reach for a SQL-parsing approach instead:<\/p>\n\n\n\n<ul><li><strong>Before the code runs on Databricks.<\/strong> You are migrating from Oracle, Teradata, or SQL Server and need the dependency graph of the legacy SQL \u2014 and of the rewritten Databricks SQL \u2014 before anything executes in production.<\/li>\n<li><strong>SQL that never touches Unity Catalog.<\/strong> Notebook SQL exported to files, scripts in a Git repo, SQL generated by external schedulers or ETL tools, code on workspaces not yet upgraded to Unity Catalog.<\/li>\n<li><strong>Cross-platform estates.<\/strong> Pipelines that start in SQL Server, transform in Databricks, and land in Snowflake need one lineage graph across all three. SQLFlow ships dialect-specific parsers for 39 databases \u2014 Databricks and <a href=\"https:\/\/www.gudusoft.com\/pt\/spark-sql-data-lineage\/\">Spark SQL<\/a> among them \u2014 so the whole chain is analyzed with one engine.<\/li>\n<li><strong>Lineage in a catalog you already run.<\/strong> If your organization standardizes on DataHub, Microsoft Purview, or OpenMetadata rather than the Databricks UI, you need lineage that exports there. SQLFlow includes export adapters for all three.<\/li><\/ul>\n\n\n\n<p>The two approaches are complementary. Unity Catalog tells you what ran; static SQL analysis tells you what the code does \u2014 including code that hasn&#8217;t run yet.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">How SQLFlow parses Databricks SQL<\/h2>\n\n\n\n<p><a href=\"https:\/\/www.gudusoft.com\/pt\/ferramenta-de-linhagem-de-dados-sql\/\">Gudu SQLFlow<\/a> is an automated SQL data lineage tool built on the General SQL Parser, a commercial SQL compiler front-end developed since the mid-2000s and validated against roughly 13,600 per-dialect test fixtures. It ships a dedicated Databricks dialect parser in the Spark SQL family \u2014 not a generic ANSI grammar \u2014 so Databricks-specific constructs are parsed as first-class syntax rather than approximated.<\/p>\n\n\n\n<p>The analysis is entirely static. SQLFlow reads SQL text and, optionally, schema metadata; it never reads the rows in your Delta tables, and with the <a href=\"https:\/\/www.gudusoft.com\/pt\/sqlflow-on-premise-versao\/\">On-Premise edition<\/a> the SQL text itself never leaves your network: it deploys on Docker or Kubernetes and runs air-gapped. For each statement, the parser builds a full semantic model, resolving every column reference through CTEs, subqueries, views, and <code>SELECT *<\/code> expansion, then the data-flow analyzer extracts source-to-target relationships at column granularity.<\/p>\n\n\n\n<p>You can feed it SQL any way it exists in your estate: pasted statements, uploaded files, database metadata over JDBC, or a dbt manifest for dbt projects targeting Databricks. Output is an interactive, drillable diagram plus structured lineage data as JSON, CSV, PNG, or a REST API response.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Example: column-level lineage through a Delta MERGE<\/h2>\n\n\n\n<p><code>MERGE INTO<\/code> is the workhorse of Delta Lake pipelines, and a statement where table-level lineage is nearly useless, because a single merge reads, matches, updates, and inserts in one shot. Consider a gold-layer upsert:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>MERGE INTO gold.customer_ltv AS t\nUSING (\n  SELECT o.customer_id,\n         SUM(o.amount)   AS lifetime_value,\n         MAX(o.order_ts) AS last_order_ts\n  FROM   silver.orders o\n  WHERE  o.status = 'completed'\n  GROUP BY o.customer_id\n) AS s\nON t.customer_id = s.customer_id\nWHEN MATCHED THEN UPDATE SET\n  t.lifetime_value = s.lifetime_value,\n  t.last_order_ts  = s.last_order_ts\nWHEN NOT MATCHED THEN INSERT\n  (customer_id, lifetime_value, last_order_ts)\n  VALUES (s.customer_id, s.lifetime_value, s.last_order_ts);<\/code><\/pre>\n\n\n\n<p>SQLFlow resolves this into precise column-level relationships. <code>gold.customer_ltv.lifetime_value<\/code> derives from <code>silver.orders.amount<\/code> through <code>SUM()<\/code>, via both the <code>UPDATE<\/code> and the <code>INSERT<\/code> branch. <code>last_order_ts<\/code> derives from <code>silver.orders.order_ts<\/code> through <code>MAX()<\/code>. The subquery alias <code>e<\/code> is resolved away; lineage points at the real source table, not the intermediate.<\/p>\n\n\n\n<p>Just as important is what SQLFlow classifies as <strong>indirect lineage<\/strong>: <code>silver.orders.status<\/code> never lands in the target, but the <code>WHERE<\/code> filter on it shapes every merged value, and <code>customer_id<\/code> drives both the <code>GROUP BY<\/code> and the match condition. SQLFlow models direct dataflow and indirect influence as distinct, separately toggleable relationship types. Most competing tools do not make that distinction, and it is exactly what you need when someone asks &#8220;will changing the <code>status<\/code> vocabulary break customer LTV?&#8221; The honest answer is yes, and only impact-aware lineage shows it.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Migrating to (or from) Databricks<\/h2>\n\n\n\n<p>Migration is one of the most common reasons teams analyze Databricks SQL outside the platform. Unity Catalog cannot help you plan a migration, because the lineage you need describes code that has never run on Databricks. A parser-based workflow does:<\/p>\n\n\n\n<ol><li><strong>Map the source estate.<\/strong> Run the legacy SQL \u2014 Oracle, Teradata, SQL Server, including stored procedures and the dynamic SQL inside them \u2014 through SQLFlow to get the true dependency graph. This tells you what to migrate first and what is dead code.<\/li>\n<li><strong>Validate the rewrite.<\/strong> Analyze the rewritten Databricks SQL with the Databricks dialect parser and compare lineage graphs. If a target column&#8217;s sources changed between the Teradata version and the Delta version, you have found a rewrite bug before it ships.<\/li>\n<li><strong>Verify nothing is orphaned.<\/strong> After cutover, batch-scan both estates \u2014 SQLFlow handles estates of 100+ databases and over a million columns, with incremental scans into a persistent lineage repository.<\/li><\/ol>\n\n\n\n<p>The same cross-dialect capability works in reverse or sideways: if part of your warehouse is moving to Snowflake instead, the <a href=\"https:\/\/www.gudusoft.com\/pt\/linhagem-de-dados-de-floco-de-neve\/\">Snowflake lineage workflow<\/a> uses the identical engine, so the two platforms end up in one comparable graph.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Exporting Databricks data lineage to DataHub, Purview, or OpenMetadata<\/h2>\n\n\n\n<p>Catalog-first platforms are good systems of record for metadata, and many organizations mandate one across all data platforms. SQLFlow slots in as the lineage engine underneath: it parses the SQL, computes column-level and indirect lineage, and pushes the result through dedicated export adapters for <strong>DataHub, Microsoft Purview, and OpenMetadata<\/strong>. JSON and CSV exports plus a REST API cover anything custom.<\/p>\n\n\n\n<p>This is also the practical answer to the multi-platform problem: lineage produced by two different extractors with two different models is hard to stitch into one graph. One parser producing one graph for all 39 dialects, exported to the catalog you already run, avoids the problem.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Ways to run it<\/h2>\n\n\n\n<figure class=\"wp-block-table\"><table><thead><tr><th>Option<\/th><th>Best for<\/th><th>Notes<\/th><\/tr><\/thead><tbody>\n<tr><td><a href=\"https:\/\/sqlflow.gudusoft.com\/?utm_source=gudusoft&amp;utm_medium=website&amp;utm_campaign=databricks-data-lineage\">Nuvem SQLFlow<\/a><\/td><td>Trying it on real Databricks SQL today<\/td><td>Free tier in the browser; premium $49.99\/month<\/td><\/tr>\n<tr><td>SQLFlow no local<\/td><td>Regulated environments; SQL must stay internal<\/td><td>Docker\/Kubernetes, air-gapped capable; $500\/month or $4,800 one-time per database type, installable on two servers<\/td><\/tr>\n<tr><td>REST API \/ CLI \/ Java library<\/td><td>Automating lineage in CI or your own platform<\/td><td>Same engine, headless; embeddable JavaScript widget with a 30+ method API for rendering diagrams in your product<\/td><\/tr>\n<\/tbody><\/table><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Frequently asked questions<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Does SQLFlow replace Unity Catalog lineage?<\/h3>\n\n\n<p>No, it complements it. Unity Catalog captures lineage for workloads that run through it, automatically. SQLFlow analyzes the SQL statically, so it covers code before it runs (migrations), SQL that executes outside Unity Catalog, and estates spanning Databricks plus other platforms, then exports the result to DataHub, Purview, or OpenMetadata.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Does SQLFlow need access to my Databricks workspace or data?<\/h3>\n\n\n<p>No. SQLFlow performs static analysis of SQL code, optionally using schema metadata to resolve references. It never reads rows in your Delta tables. With the On-Premise edition, even the SQL text stays inside your network.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Can SQLFlow trace lineage through MERGE INTO statements?<\/h3>\n\n\n<p>Yes. For a Delta <code>MERGE INTO<\/code>, SQLFlow resolves the <code>USING<\/code> subquery, maps each target column to its true source columns through both the <code>UPDATE<\/code> e <code>INSERT<\/code> branches, and classifies match conditions and filters as indirect (impact) lineage.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Is Databricks a distinct dialect from Spark SQL in SQLFlow?<\/h3>\n\n\n<p>Yes. SQLFlow&#8217;s 39 supported dialects list Databricks and Spark SQL separately, each with its own parser in the Spark SQL family. Pick Databricks for SQL written against Databricks; there is also a dedicated <a href=\"https:\/\/www.gudusoft.com\/pt\/spark-sql-data-lineage\/\">Spark SQL lineage page<\/a> for open-source Spark workloads.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Does it work with dbt projects on Databricks?<\/h3>\n\n\n<p>Yes. Import the dbt manifest and SQLFlow produces column-level lineage across your models, using the Databricks dialect to parse the compiled SQL.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">What does SQLFlow cost?<\/h3>\n\n\n<p>SQLFlow Cloud starts free; premium is $49.99\/month. On-Premise is $500\/month or $4,800 one-time per selected database type. Full details are on the <a href=\"https:\/\/www.gudusoft.com\/pt\/precos\/\">pricing page<\/a>.<\/p>\n\n\n\n<div class=\"wp-block-group alignfull has-background is-layout-constrained\" style=\"background-color:#60d5f6;padding-top:32px;padding-bottom:32px\"><div class=\"wp-block-group__inner-container\">\n\n<h2 class=\"wp-block-heading\">Trace your Databricks lineage now<\/h2>\n\n\n<p>Paste a Spark SQL query or Delta MERGE into the free visualizer, or talk to us about scanning a whole migration estate.<\/p>\n\n\n<div class=\"wp-block-buttons is-layout-flex\">\n<div class=\"wp-block-button\"><a class=\"wp-block-button__link wp-element-button\" href=\"https:\/\/sqlflow.gudusoft.com\/?utm_source=gudusoft&amp;utm_medium=website&amp;utm_campaign=databricks-data-lineage\" target=\"_blank\" rel=\"noreferrer noopener\">Try SQLFlow free<\/a><\/div>\n<\/div>\n\n\n<div class=\"wp-block-buttons is-layout-flex\">\n<div class=\"wp-block-button\"><a class=\"wp-block-button__link wp-element-button\" href=\"https:\/\/www.gudusoft.com\/pt\/contato\/\">Request an enterprise demo<\/a><\/div>\n<\/div>\n\n<\/div><\/div>\n\n\n\n<script type=\"application\/ld+json\">{\n    \"@context\": \"https:\\\/\\\/schema.org\",\n    \"@graph\": [\n        {\n            \"@type\": \"SoftwareApplication\",\n            \"name\": \"Gudu SQLFlow\",\n            \"applicationCategory\": \"DeveloperApplication\",\n            \"applicationSubCategory\": \"SQL Data Lineage Tool\",\n            \"operatingSystem\": \"Web, Linux, Windows, macOS\",\n            \"url\": \"https:\\\/\\\/www.gudusoft.com\\\/sql-data-lineage-tool\\\/\",\n            \"description\": \"Automated SQL data lineage tool with a dedicated Databricks dialect parser: column-level lineage for Spark SQL and Delta Lake MERGE statements, with export to DataHub, Microsoft Purview, and OpenMetadata.\",\n            \"featureList\": \"Databricks dialect parser, column-level lineage, indirect\\\/impact lineage, Delta MERGE INTO analysis, dbt support, cross-platform migration analysis across 39 dialects, DataHub\\\/Purview\\\/OpenMetadata export, REST API\",\n            \"softwareVersion\": \"8.2.3\",\n            \"offers\": [\n                {\n                    \"@type\": \"Offer\",\n                    \"name\": \"SQLFlow Cloud Free\",\n                    \"price\": \"0\",\n                    \"priceCurrency\": \"USD\"\n                },\n                {\n                    \"@type\": \"Offer\",\n                    \"name\": \"SQLFlow Cloud Premium\",\n                    \"price\": \"49.99\",\n                    \"priceCurrency\": \"USD\",\n                    \"priceSpecification\": {\n                        \"@type\": \"UnitPriceSpecification\",\n                        \"price\": \"49.99\",\n                        \"priceCurrency\": \"USD\",\n                        \"billingIncrement\": 1,\n                        \"unitText\": \"MONTH\"\n                    }\n                },\n                {\n                    \"@type\": \"Offer\",\n                    \"name\": \"SQLFlow On-Premise\",\n                    \"price\": \"4800\",\n                    \"priceCurrency\": \"USD\"\n                }\n            ],\n            \"publisher\": {\n                \"@type\": \"Organization\",\n                \"name\": \"Gudu Software\",\n                \"url\": \"https:\\\/\\\/www.gudusoft.com\\\/\"\n            }\n        },\n        {\n            \"@type\": \"FAQPage\",\n            \"mainEntity\": [\n                {\n                    \"@type\": \"Question\",\n                    \"name\": \"Does SQLFlow replace Unity Catalog lineage?\",\n                    \"acceptedAnswer\": {\n                        \"@type\": \"Answer\",\n                        \"text\": \"No, it complements it. Unity Catalog captures lineage for workloads that run through it. SQLFlow analyzes SQL statically, covering code before it runs (migrations), SQL executed outside Unity Catalog, and cross-platform estates, then exports to DataHub, Purview, or OpenMetadata.\"\n                    }\n                },\n                {\n                    \"@type\": \"Question\",\n                    \"name\": \"Does SQLFlow need access to my Databricks workspace or data?\",\n                    \"acceptedAnswer\": {\n                        \"@type\": \"Answer\",\n                        \"text\": \"No. SQLFlow performs static analysis of SQL code and optionally reads schema metadata. It never reads rows in your Delta tables, and the On-Premise edition keeps SQL text inside your network.\"\n                    }\n                },\n                {\n                    \"@type\": \"Question\",\n                    \"name\": \"Can SQLFlow trace lineage through MERGE INTO statements?\",\n                    \"acceptedAnswer\": {\n                        \"@type\": \"Answer\",\n                        \"text\": \"Yes. For a Delta MERGE INTO, SQLFlow resolves the USING subquery, maps each target column to its source columns through both the UPDATE and INSERT branches, and classifies match conditions and filters as indirect lineage.\"\n                    }\n                },\n                {\n                    \"@type\": \"Question\",\n                    \"name\": \"Is Databricks a distinct dialect from Spark SQL in SQLFlow?\",\n                    \"acceptedAnswer\": {\n                        \"@type\": \"Answer\",\n                        \"text\": \"Yes. SQLFlow's 39 supported dialects list Databricks and Spark SQL separately, each with its own dialect-specific parser in the Spark SQL family.\"\n                    }\n                },\n                {\n                    \"@type\": \"Question\",\n                    \"name\": \"Does SQLFlow work with dbt projects on Databricks?\",\n                    \"acceptedAnswer\": {\n                        \"@type\": \"Answer\",\n                        \"text\": \"Yes. Import the dbt manifest and SQLFlow produces column-level lineage across your models, parsing the compiled SQL with the Databricks dialect.\"\n                    }\n                },\n                {\n                    \"@type\": \"Question\",\n                    \"name\": \"What does SQLFlow cost?\",\n                    \"acceptedAnswer\": {\n                        \"@type\": \"Answer\",\n                        \"text\": \"SQLFlow Cloud starts free; premium is $49.99\\\/month. SQLFlow On-Premise is $500\\\/month or $4,800 one-time per selected database type.\"\n                    }\n                }\n            ]\n        }\n    ]\n}<\/script>","protected":false},"excerpt":{"rendered":"<p>Databricks data lineage is the column-level map of how data flows through your Spark SQL and Delta Lake code: which source columns feed each target table across your bronze, silver, and gold layers, and which transformations \u2014 joins, aggregates, MERGE logic \u2014 happen along the way. Unity Catalog captures lineage automatically for workloads that run through it; Gudu SQLFlow covers everything else by parsing the SQL itself, which makes it the tool for migrations into Databricks, SQL that runs outside Unity Catalog, and estates that span Databricks plus other platforms. Try it in 30 seconds: paste a Databricks MERGE INTO\u2026<\/p>","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":[],"blocksy_meta":{"styles_descriptor":{"styles":{"desktop":"","tablet":"","mobile":""},"google_fonts":[],"version":5}},"_links":{"self":[{"href":"https:\/\/www.gudusoft.com\/pt\/wp-json\/wp\/v2\/pages\/6753"}],"collection":[{"href":"https:\/\/www.gudusoft.com\/pt\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.gudusoft.com\/pt\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.gudusoft.com\/pt\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.gudusoft.com\/pt\/wp-json\/wp\/v2\/comments?post=6753"}],"version-history":[{"count":0,"href":"https:\/\/www.gudusoft.com\/pt\/wp-json\/wp\/v2\/pages\/6753\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.gudusoft.com\/pt\/wp-json\/wp\/v2\/media?parent=6753"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}