{"id":6761,"date":"2026-07-12T02:07:41","date_gmt":"2026-07-12T10:07:41","guid":{"rendered":"https:\/\/www.gudusoft.com\/spark-sql-data-lineage\/"},"modified":"2026-07-12T02:07:41","modified_gmt":"2026-07-12T10:07:41","slug":"spark-sql-data-lineage","status":"publish","type":"page","link":"https:\/\/www.gudusoft.com\/fr\/spark-sql-data-lineage\/","title":{"rendered":"Spark SQL Data Lineage: Column Lineage for Spark Pipelines"},"content":{"rendered":"<p><strong>Spark SQL data lineage<\/strong> is the column-level map of how data moves through your Spark SQL code: which source columns feed every table written by a <code>CREATE TABLE ... AS SELECT<\/code> ou <code>INSERT OVERWRITE<\/code>, and through which temp views, joins, functions, and filters along the way. <strong>Gudu SQLFlow<\/strong> builds that map by statically parsing your Spark SQL with a dedicated Spark dialect parser \u2014 no cluster to instrument, no listener to attach, and no job run required. Paste the SQL, get the lineage diagram.<\/p>\n\n\n\n<div class=\"wp-block-group alignfull has-background is-layout-constrained\" style=\"background-color:#eef7fb;padding-top:24px;padding-bottom:24px\"><div class=\"wp-block-group__inner-container\">\n\n<p><strong>Essayez-le maintenant\u00a0:<\/strong> paste any Spark SQL script into the <a href=\"https:\/\/sqlflow.gudusoft.com\/?utm_source=gudusoft&amp;utm_medium=website&amp;utm_campaign=spark-sql-data-lineage\" target=\"_blank\" rel=\"noreferrer noopener\">Visualiseur de lign\u00e9es SQL en ligne gratuit<\/a>, select the Spark dialect, and trace any column upstream or downstream. The Cloud edition has a free tier.<\/p>\n\n<\/div><\/div>\n\n\n\n<h2 class=\"wp-block-heading\">Runtime listeners vs. static parsing: two ways to get Spark lineage<\/h2>\n\n\n\n<p>There are two fundamentally different approaches to lineage in the Spark world, and most teams eventually need both.<\/p>\n\n\n\n<p><strong>Runtime lineage<\/strong> \u2014 the approach taken by OpenLineage and its Spark integration \u2014 attaches a listener to the Spark session and emits lineage events as each job executes. It is genuinely good at what it does: it records what actually ran, when it ran, and against which datasets, with run-level metadata that static analysis cannot know. If your question is &#8220;what did last night&#8217;s pipeline actually touch?&#8221;, runtime lineage answers it.<\/p>\n\n\n\n<p><strong>Static lineage<\/strong> \u2014 SQLFlow&#8217;s approach \u2014 parses the SQL text itself and derives lineage from the code, the way a compiler would. That covers everything runtime capture structurally cannot:<\/p>\n\n\n\n<ul><li><strong>SQL you haven&#8217;t run yet.<\/strong> A new pipeline in a pull request, a script inherited from another team, a backlog of jobs scheduled quarterly \u2014 none of these emit runtime events until they execute, possibly against production data.<\/li>\n<li><strong>Code review.<\/strong> Reviewers can see the full column-level dependency graph of a change before approving it, instead of discovering the impact after deployment.<\/li>\n<li><strong>Migration audits.<\/strong> When you move workloads onto Spark \u2014 commonly from Hive \u2014 you need the dependency graph of the <em>entire<\/em> codebase, including jobs that only run at month-end. Static parsing reads it all in one pass.<\/li><\/ul>\n\n\n\n<figure class=\"wp-block-table\"><table><thead><tr><th><\/th><th>Runtime listeners (OpenLineage)<\/th><th>Static parsing (SQLFlow)<\/th><\/tr><\/thead><tbody>\n<tr><td>Lineage captured<\/td><td>Per job run, as it executes<\/td><td>From SQL text, before anything runs<\/td><\/tr>\n<tr><td>Requires<\/td><td>Listener configured on the Spark session; jobs must execute<\/td><td>The SQL files, plus optional schema metadata<\/td><\/tr>\n<tr><td>Covers unexecuted code<\/td><td>No \u2014 no run, no event<\/td><td>Yes \u2014 PRs, backlogs, rarely-run jobs<\/td><\/tr>\n<tr><td>Run metadata (timing, versions)<\/td><td>Yes<\/td><td>No \u2014 code-level only<\/td><\/tr>\n<tr><td>Typical use<\/td><td>Operational observability of live pipelines<\/td><td>Impact analysis, code review, migration audits, compliance documentation<\/td><\/tr>\n<\/tbody><\/table><\/figure>\n\n\n\n<p>These are complementary, not competing. Runtime lineage tells you what happened; static lineage tells you what the code does. SQLFlow&#8217;s export adapters for DataHub, Microsoft Purview, and OpenMetadata let you land statically-derived lineage in the same catalog where your runtime events go, so the two views sit side by side.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Example: a temp view chain into a CTAS<\/h2>\n\n\n\n<p>The pattern that defeats naive lineage tools in Spark SQL is the temp view chain. Notebooks and jobs routinely stage logic through <code>CREATE OR REPLACE TEMP VIEW<\/code> steps before a final <code>CREATE TABLE ... AS SELECT<\/code>. Here is a compact but realistic example:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>CREATE OR REPLACE TEMP VIEW clean_events AS\nSELECT\n  CAST(event_time AS TIMESTAMP) AS event_ts,\n  user_id,\n  lower(event_name)             AS event_name,\n  amount\nFROM raw.events\nWHERE event_date &gt;= date_sub(current_date(), 30);\n\nCREATE OR REPLACE TEMP VIEW user_daily AS\nSELECT\n  user_id,\n  to_date(event_ts)  AS event_day,\n  SUM(amount)        AS daily_spend\nFROM clean_events\nWHERE event_name = 'purchase'\nGROUP BY user_id, to_date(event_ts);\n\nCREATE TABLE analytics.user_spend_30d\nUSING PARQUET\nAS\nSELECT\n  u.user_id,\n  u.segment,\n  d.event_day,\n  d.daily_spend\nFROM user_daily d\nJOIN dim.users u ON u.user_id = d.user_id;<\/code><\/pre>\n\n\n\n<p>Ask &#8220;where does <code>analytics.user_spend_30d.daily_spend<\/code> come from?&#8221; and the honest answer requires resolving two view layers: <code>daily_spend<\/code> est <code>SUM(clean_events.amount)<\/code>, et <code>clean_events.amount<\/code> est <code>raw.events.amount<\/code>. SQLFlow performs exactly that resolution and renders the full chain \u2014 <code>raw.events.amount<\/code> \u00e0 travers <code>clean_events<\/code> et <code>user_daily<\/code> into the final table, with the <code>SOMME<\/code> aggregation attached to the edge where it happens.<\/p>\n\n\n\n<p>It also captures what most tools drop entirely: <strong>lign\u00e9e indirecte<\/strong>. The columns <code>event_date<\/code>, <code>event_name<\/code>, and the join key <code>user_id<\/code> never appear in <code>daily_spend<\/code>, yet all three shape its value \u2014 change the 30-day filter or the <code>'purchase'<\/code> predicate and every number in the output changes. SQLFlow models these as a distinct, toggleable relationship type, so you can view pure dataflow or the full impact surface. For a schema-change review, the impact surface is the view you actually want.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Spark SQL constructs SQLFlow resolves<\/h2>\n\n\n\n<p>SQLFlow ships a Spark SQL dialect parser \u2014 one of 39 dialect-specific parsers, not a generic ANSI grammar with Spark keywords bolted on. On Spark SQL specifically, it handles the statements that carry lineage:<\/p>\n\n\n\n<ul><li><strong><code>CREATE TABLE ... AS SELECT<\/code> (CTAS)<\/strong> \u2014 the workhorse of Spark batch pipelines; every output column is traced to its sources through the full SELECT.<\/li>\n<li><strong><code>INSERT INTO<\/code> et <code>INSERT OVERWRITE<\/code><\/strong> \u2014 including the positional column mapping between the SELECT list and the target table&#8217;s schema.<\/li>\n<li><strong><code>CREATE OR REPLACE TEMP VIEW<\/code><\/strong> \u2014 temp views are resolved and chained, so lineage flows through staging layers instead of stopping at them.<\/li>\n<li><strong>CTEs, subqueries, and <code>S\u00c9LECTIONNER *<\/code><\/strong> \u2014 column references are resolved through common table expressions and nested subqueries, and star expressions are expanded to real columns when schema metadata is available.<\/li>\n<li><strong>Functions, casts, and set operators<\/strong> \u2014 each output column&#8217;s lineage records which transformations it passed through, so <code>daily_spend<\/code> is not just &#8220;from amount&#8221; but &#8220;from <code>montant<\/code> via <code>SOMME<\/code>&#8220;.<\/li><\/ul>\n\n\n\n<p>Under the hood this is the General SQL Parser engine \u2014 a commercial SQL compiler front-end developed since the mid-2000s and validated against roughly 13,600 per-dialect test fixtures. Lineage quality is a parsing problem before it is anything else, and that corpus is what keeps edge-case Spark syntax from silently producing wrong edges.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Where Spark SQL data lineage fits in your stack<\/h2>\n\n\n\n<p>Spark rarely lives alone. Its tables usually sit in a Hive metastore, and an increasing share of Spark workloads run on Databricks. SQLFlow treats these as first-class, separately parsed dialects:<\/p>\n\n\n\n<ul><li>Migrating Hive jobs to Spark? Analyze the legacy HQL with the <a href=\"https:\/\/www.gudusoft.com\/fr\/lignee-des-donnees-de-la-ruche\/\">Hive data lineage parser<\/a> and the rewritten jobs with the Spark parser, and compare the two dependency graphs to verify nothing was dropped in translation.<\/li>\n<li>Running on Databricks? The <a href=\"https:\/\/www.gudusoft.com\/fr\/lignee-des-donnees-databricks\/\">Databricks data lineage parser<\/a> covers the Databricks SQL dialect, which has diverged from open-source Spark SQL in ways a single shared grammar mishandles.<\/li>\n<li>Mixed estate? SQLFlow analyzes all 39 supported dialects into one lineage repository, so a Spark job reading a table produced by a Snowflake task shows up as one continuous graph. See the <a href=\"https:\/\/www.gudusoft.com\/fr\/outil-de-lignee-de-donnees-sql\/\">Pr\u00e9sentation de l&#039;outil de tra\u00e7abilit\u00e9 des donn\u00e9es SQL<\/a> for the full picture.<\/li><\/ul>\n\n\n\n<p>Inputs are flexible: paste SQL, upload files, pull metadata over JDBC, or import a dbt manifest for dbt-on-Spark projects. Output is an interactive drillable diagram plus JSON, CSV, and PNG exports and a REST API for automation \u2014 for example, analyzing the SQL changed in a pull request as a CI step.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Deployment and scale<\/h2>\n\n\n\n<p>SQLFlow Cloud has a free tier for interactive use; premium is $49.99\/month. Teams whose SQL cannot leave the network run <a href=\"https:\/\/www.gudusoft.com\/fr\/sqlflow-on-premise-version\/\">SQLFlow sur site<\/a> \u2014 Docker or Kubernetes, air-gap friendly, $500\/month or $4,800 one-time per selected database type. Either way, SQLFlow performs static analysis of SQL code only: it never reads the rows in your tables. Enterprise deployments batch-scan estates of 100+ databases and over a million columns with incremental scans and a persistent lineage repository.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Foire aux questions<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Do I need a running Spark cluster to get lineage?<\/h3>\n\n\n<p>No. SQLFlow parses the Spark SQL text statically. You can analyze a script that has never executed anywhere \u2014 a pull request, a vendor delivery, a migration backlog \u2014 and get the same column-level lineage you would get from production code.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">How is this different from OpenLineage&#8217;s Spark integration?<\/h3>\n\n\n<p>OpenLineage captures lineage at run time via a listener on the Spark session \u2014 excellent for observing what live jobs actually did. SQLFlow derives lineage from the SQL code itself, so it also covers code that hasn&#8217;t run, supports pre-merge review and migration audits, and needs no cluster access. The two are complementary; SQLFlow can export into DataHub, Purview, or OpenMetadata alongside runtime lineage.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Can SQLFlow trace lineage through temp views?<\/h3>\n\n\n<p>Yes. Chains of <code>CREATE OR REPLACE TEMP VIEW<\/code> statements are resolved end to end, so a column in a final CTAS traces back through every staging view to the physical source table, with the functions and aggregations applied at each step.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Is Databricks SQL the same as Spark SQL here?<\/h3>\n\n\n<p>No \u2014 SQLFlow parses them with separate dialect parsers. Databricks SQL has its own syntax extensions, so it gets its own grammar. If you run on Databricks, use the Databricks dialect; for open-source Spark, EMR, or self-managed clusters, use the Spark SQL dialect.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Does SQLFlow read my data?<\/h3>\n\n\n<p>No. It analyzes SQL code and, optionally, schema metadata (table and column definitions). Row data is never accessed. With On-Premise, even the SQL text stays inside your network.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">What does Spark SQL lineage with SQLFlow cost?<\/h3>\n\n\n<p>SQLFlow Cloud starts free; premium is $49.99\/month. On-Premise is $500\/month or $4,800 one-time per selected database type, installable on two servers. See <a href=\"https:\/\/www.gudusoft.com\/fr\/tarification\/\">tarification<\/a> pour plus de d\u00e9tails.<\/p>\n\n\n\n<div class=\"wp-block-group alignfull has-background is-layout-constrained\" style=\"background-color:#60d5f6;padding-top:32px;padding-bottom:32px\"><div class=\"wp-block-group__inner-container\">\n\n<h2 class=\"wp-block-heading\">Trace your Spark SQL now<\/h2>\n\n\n<p>Paste a Spark SQL script into the free visualizer and follow any column from source to target \u2014 or talk to us about scanning your whole estate.<\/p>\n\n\n<div class=\"wp-block-buttons is-layout-flex\">\n<div class=\"wp-block-button\"><a class=\"wp-block-button__link wp-element-button\" href=\"https:\/\/sqlflow.gudusoft.com\/?utm_source=gudusoft&amp;utm_medium=website&amp;utm_campaign=spark-sql-data-lineage\" target=\"_blank\" rel=\"noreferrer noopener\">Essayez SQLFlow gratuitement<\/a><\/div>\n<\/div>\n\n\n<div class=\"wp-block-buttons is-layout-flex\">\n<div class=\"wp-block-button\"><a class=\"wp-block-button__link wp-element-button\" href=\"https:\/\/www.gudusoft.com\/fr\/contact\/\">Demander une d\u00e9monstration pour entreprises<\/a><\/div>\n<\/div>\n\n<\/div><\/div>\n\n\n\n<script type=\"application\/ld+json\">{\n    \"@context\": \"https:\\\/\\\/schema.org\",\n    \"@graph\": [\n        {\n            \"@type\": \"SoftwareApplication\",\n            \"name\": \"Gudu SQLFlow\",\n            \"applicationCategory\": \"DeveloperApplication\",\n            \"applicationSubCategory\": \"SQL Data Lineage Tool\",\n            \"operatingSystem\": \"Web, Linux, Windows, macOS\",\n            \"url\": \"https:\\\/\\\/www.gudusoft.com\\\/spark-sql-data-lineage\\\/\",\n            \"description\": \"Automated Spark SQL data lineage from static parsing: column-level lineage through CTAS, INSERT INTO\\\/OVERWRITE, and temp view chains, with no cluster or job run required.\",\n            \"featureList\": \"Spark SQL dialect parser, column-level lineage, temp view resolution, CTAS and INSERT OVERWRITE analysis, indirect\\\/impact lineage, dbt support, REST API, DataHub\\\/Purview\\\/OpenMetadata export\",\n            \"softwareVersion\": \"8.2.3\",\n            \"offers\": [\n                {\n                    \"@type\": \"Offer\",\n                    \"name\": \"SQLFlow Cloud Free\",\n                    \"price\": \"0\",\n                    \"priceCurrency\": \"USD\"\n                },\n                {\n                    \"@type\": \"Offer\",\n                    \"name\": \"SQLFlow Cloud Premium\",\n                    \"price\": \"49.99\",\n                    \"priceCurrency\": \"USD\",\n                    \"priceSpecification\": {\n                        \"@type\": \"UnitPriceSpecification\",\n                        \"price\": \"49.99\",\n                        \"priceCurrency\": \"USD\",\n                        \"billingIncrement\": 1,\n                        \"unitText\": \"MONTH\"\n                    }\n                },\n                {\n                    \"@type\": \"Offer\",\n                    \"name\": \"SQLFlow On-Premise\",\n                    \"price\": \"4800\",\n                    \"priceCurrency\": \"USD\"\n                }\n            ],\n            \"publisher\": {\n                \"@type\": \"Organization\",\n                \"name\": \"Gudu Software\",\n                \"url\": \"https:\\\/\\\/www.gudusoft.com\\\/\"\n            }\n        },\n        {\n            \"@type\": \"FAQPage\",\n            \"mainEntity\": [\n                {\n                    \"@type\": \"Question\",\n                    \"name\": \"Do I need a running Spark cluster to get lineage?\",\n                    \"acceptedAnswer\": {\n                        \"@type\": \"Answer\",\n                        \"text\": \"No. SQLFlow parses the Spark SQL text statically. You can analyze a script that has never executed anywhere and get the same column-level lineage you would get from production code.\"\n                    }\n                },\n                {\n                    \"@type\": \"Question\",\n                    \"name\": \"How is this different from OpenLineage's Spark integration?\",\n                    \"acceptedAnswer\": {\n                        \"@type\": \"Answer\",\n                        \"text\": \"OpenLineage captures lineage at run time via a listener on the Spark session, which is excellent for observing what live jobs actually did. SQLFlow derives lineage from the SQL code itself, so it also covers code that hasn't run, supports pre-merge review and migration audits, and needs no cluster access. The two are complementary.\"\n                    }\n                },\n                {\n                    \"@type\": \"Question\",\n                    \"name\": \"Can SQLFlow trace lineage through temp views?\",\n                    \"acceptedAnswer\": {\n                        \"@type\": \"Answer\",\n                        \"text\": \"Yes. Chains of CREATE OR REPLACE TEMP VIEW statements are resolved end to end, so a column in a final CTAS traces back through every staging view to the physical source table, with the functions and aggregations applied at each step.\"\n                    }\n                },\n                {\n                    \"@type\": \"Question\",\n                    \"name\": \"Is Databricks SQL the same as Spark SQL here?\",\n                    \"acceptedAnswer\": {\n                        \"@type\": \"Answer\",\n                        \"text\": \"No. SQLFlow parses them with separate dialect parsers. Databricks SQL has its own syntax extensions, so it gets its own grammar. Use the Databricks dialect on Databricks and the Spark SQL dialect for open-source Spark, EMR, or self-managed clusters.\"\n                    }\n                },\n                {\n                    \"@type\": \"Question\",\n                    \"name\": \"Does SQLFlow read my data?\",\n                    \"acceptedAnswer\": {\n                        \"@type\": \"Answer\",\n                        \"text\": \"No. It analyzes SQL code and, optionally, schema metadata such as table and column definitions. Row data is never accessed. With the On-Premise edition, even the SQL text stays inside your network.\"\n                    }\n                },\n                {\n                    \"@type\": \"Question\",\n                    \"name\": \"What does Spark SQL lineage with SQLFlow cost?\",\n                    \"acceptedAnswer\": {\n                        \"@type\": \"Answer\",\n                        \"text\": \"SQLFlow Cloud starts free; premium is $49.99\\\/month. SQLFlow On-Premise is $500\\\/month or $4,800 one-time per selected database type, installable on two servers.\"\n                    }\n                }\n            ]\n        }\n    ]\n}<\/script>","protected":false},"excerpt":{"rendered":"<p>Spark SQL data lineage is the column-level map of how data moves through your Spark SQL code: which source columns feed every table written by a CREATE TABLE &#8230; AS SELECT or INSERT OVERWRITE, and through which temp views, joins, functions, and filters along the way. Gudu SQLFlow builds that map by statically parsing your Spark SQL with a dedicated Spark dialect parser \u2014 no cluster to instrument, no listener to attach, and no job run required. Paste the SQL, get the lineage diagram. Try it now: paste any Spark SQL script into the free online SQL lineage visualizer, select\u2026<\/p>","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":[],"blocksy_meta":{"styles_descriptor":{"styles":{"desktop":"","tablet":"","mobile":""},"google_fonts":[],"version":5}},"_links":{"self":[{"href":"https:\/\/www.gudusoft.com\/fr\/wp-json\/wp\/v2\/pages\/6761"}],"collection":[{"href":"https:\/\/www.gudusoft.com\/fr\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.gudusoft.com\/fr\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.gudusoft.com\/fr\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.gudusoft.com\/fr\/wp-json\/wp\/v2\/comments?post=6761"}],"version-history":[{"count":0,"href":"https:\/\/www.gudusoft.com\/fr\/wp-json\/wp\/v2\/pages\/6761\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.gudusoft.com\/fr\/wp-json\/wp\/v2\/media?parent=6761"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}