Spark SQL Data Lineage: Column Lineage for Spark Pipelines

Spark SQL data lineage is the column-level map of how data moves through your Spark SQL code: which source columns feed every table written by a CREATE TABLE ... AS SELECT o INSERT OVERWRITE, and through which temp views, joins, functions, and filters along the way. Flujo de SQL de Gudu builds that map by statically parsing your Spark SQL with a dedicated Spark dialect parser — no cluster to instrument, no listener to attach, and no job run required. Paste the SQL, get the lineage diagram.

Runtime listeners vs. static parsing: two ways to get Spark lineage

There are two fundamentally different approaches to lineage in the Spark world, and most teams eventually need both.

Runtime lineage — the approach taken by OpenLineage and its Spark integration — attaches a listener to the Spark session and emits lineage events as each job executes. It is genuinely good at what it does: it records what actually ran, when it ran, and against which datasets, with run-level metadata that static analysis cannot know. If your question is “what did last night’s pipeline actually touch?”, runtime lineage answers it.

Static lineage — SQLFlow’s approach — parses the SQL text itself and derives lineage from the code, the way a compiler would. That covers everything runtime capture structurally cannot:

• SQL you haven’t run yet. A new pipeline in a pull request, a script inherited from another team, a backlog of jobs scheduled quarterly — none of these emit runtime events until they execute, possibly against production data.
• Code review. Reviewers can see the full column-level dependency graph of a change before approving it, instead of discovering the impact after deployment.
• Migration audits. When you move workloads onto Spark — commonly from Hive — you need the dependency graph of the entire codebase, including jobs that only run at month-end. Static parsing reads it all in one pass.

	Runtime listeners (OpenLineage)	Static parsing (SQLFlow)
Lineage captured	Per job run, as it executes	From SQL text, before anything runs
Requires	Listener configured on the Spark session; jobs must execute	The SQL files, plus optional schema metadata
Covers unexecuted code	No — no run, no event	Yes — PRs, backlogs, rarely-run jobs
Run metadata (timing, versions)	Yes	No — code-level only
Typical use	Operational observability of live pipelines	Impact analysis, code review, migration audits, compliance documentation

These are complementary, not competing. Runtime lineage tells you what happened; static lineage tells you what the code does. SQLFlow’s export adapters for DataHub, Microsoft Purview, and OpenMetadata let you land statically-derived lineage in the same catalog where your runtime events go, so the two views sit side by side.

Example: a temp view chain into a CTAS

The pattern that defeats naive lineage tools in Spark SQL is the temp view chain. Notebooks and jobs routinely stage logic through CREATE OR REPLACE TEMP VIEW steps before a final CREATE TABLE ... AS SELECT. Here is a compact but realistic example:

CREATE OR REPLACE TEMP VIEW clean_events AS
SELECT
  CAST(event_time AS TIMESTAMP) AS event_ts,
  user_id,
  lower(event_name)             AS event_name,
  amount
FROM raw.events
WHERE event_date >= date_sub(current_date(), 30);

CREATE OR REPLACE TEMP VIEW user_daily AS
SELECT
  user_id,
  to_date(event_ts)  AS event_day,
  SUM(amount)        AS daily_spend
FROM clean_events
WHERE event_name = 'purchase'
GROUP BY user_id, to_date(event_ts);

CREATE TABLE analytics.user_spend_30d
USING PARQUET
AS
SELECT
  u.user_id,
  u.segment,
  d.event_day,
  d.daily_spend
FROM user_daily d
JOIN dim.users u ON u.user_id = d.user_id;

Ask “where does analytics.user_spend_30d.daily_spend come from?” and the honest answer requires resolving two view layers: daily_spend is SUM(clean_events.amount), y clean_events.amount is raw.events.amount. SQLFlow performs exactly that resolution and renders the full chain — raw.events.amount through clean_events y user_daily into the final table, with the SUM aggregation attached to the edge where it happens.

It also captures what most tools drop entirely: indirect lineage. The columns event_date, event_name, and the join key user_id never appear in daily_spend, yet all three shape its value — change the 30-day filter or the 'purchase' predicate and every number in the output changes. SQLFlow models these as a distinct, toggleable relationship type, so you can view pure dataflow or the full impact surface. For a schema-change review, the impact surface is the view you actually want.

Spark SQL constructs SQLFlow resolves

SQLFlow ships a Spark SQL dialect parser — one of 39 dialect-specific parsers, not a generic ANSI grammar with Spark keywords bolted on. On Spark SQL specifically, it handles the statements that carry lineage:

• CREATE TABLE ... AS SELECT (CTAS) — the workhorse of Spark batch pipelines; every output column is traced to its sources through the full SELECT.
• INSERT INTO y INSERT OVERWRITE — including the positional column mapping between the SELECT list and the target table’s schema.
• CREATE OR REPLACE TEMP VIEW — temp views are resolved and chained, so lineage flows through staging layers instead of stopping at them.
• CTEs, subqueries, and SELECT * — column references are resolved through common table expressions and nested subqueries, and star expressions are expanded to real columns when schema metadata is available.
• Functions, casts, and set operators — each output column’s lineage records which transformations it passed through, so daily_spend is not just “from amount” but “from amount via SUM“.

Under the hood this is the General SQL Parser engine — a commercial SQL compiler front-end developed since the mid-2000s and validated against roughly 13,600 per-dialect test fixtures. Lineage quality is a parsing problem before it is anything else, and that corpus is what keeps edge-case Spark syntax from silently producing wrong edges.

Where Spark SQL data lineage fits in your stack

Spark rarely lives alone. Its tables usually sit in a Hive metastore, and an increasing share of Spark workloads run on Databricks. SQLFlow treats these as first-class, separately parsed dialects:

• Migrating Hive jobs to Spark? Analyze the legacy HQL with the Hive data lineage parser and the rewritten jobs with the Spark parser, and compare the two dependency graphs to verify nothing was dropped in translation.
• Running on Databricks? The Databricks data lineage parser covers the Databricks SQL dialect, which has diverged from open-source Spark SQL in ways a single shared grammar mishandles.
• Mixed estate? SQLFlow analyzes all 39 supported dialects into one lineage repository, so a Spark job reading a table produced by a Snowflake task shows up as one continuous graph. See the SQL data lineage tool overview for the full picture.

Inputs are flexible: paste SQL, upload files, pull metadata over JDBC, or import a dbt manifest for dbt-on-Spark projects. Output is an interactive drillable diagram plus JSON, CSV, and PNG exports and a REST API for automation — for example, analyzing the SQL changed in a pull request as a CI step.

Deployment and scale

SQLFlow Cloud has a free tier for interactive use; premium is $49.99/month. Teams whose SQL cannot leave the network run SQLFlow local — Docker or Kubernetes, air-gap friendly, $500/month or $4,800 one-time per selected database type. Either way, SQLFlow performs static analysis of SQL code only: it never reads the rows in your tables. Enterprise deployments batch-scan estates of 100+ databases and over a million columns with incremental scans and a persistent lineage repository.

Frequently asked questions

Do I need a running Spark cluster to get lineage?

No. SQLFlow parses the Spark SQL text statically. You can analyze a script that has never executed anywhere — a pull request, a vendor delivery, a migration backlog — and get the same column-level lineage you would get from production code.

How is this different from OpenLineage’s Spark integration?

OpenLineage captures lineage at run time via a listener on the Spark session — excellent for observing what live jobs actually did. SQLFlow derives lineage from the SQL code itself, so it also covers code that hasn’t run, supports pre-merge review and migration audits, and needs no cluster access. The two are complementary; SQLFlow can export into DataHub, Purview, or OpenMetadata alongside runtime lineage.

Can SQLFlow trace lineage through temp views?

Yes. Chains of CREATE OR REPLACE TEMP VIEW statements are resolved end to end, so a column in a final CTAS traces back through every staging view to the physical source table, with the functions and aggregations applied at each step.

Is Databricks SQL the same as Spark SQL here?

No — SQLFlow parses them with separate dialect parsers. Databricks SQL has its own syntax extensions, so it gets its own grammar. If you run on Databricks, use the Databricks dialect; for open-source Spark, EMR, or self-managed clusters, use the Spark SQL dialect.

Does SQLFlow read my data?

No. It analyzes SQL code and, optionally, schema metadata (table and column definitions). Row data is never accessed. With On-Premise, even the SQL text stays inside your network.

What does Spark SQL lineage with SQLFlow cost?

SQLFlow Cloud starts free; premium is $49.99/month. On-Premise is $500/month or $4,800 one-time per selected database type, installable on two servers. See precios for details.