DuckDB Data Lineage: Column-Level Lineage for In-Process Analytics

DuckDB data lineage is the column-level map of how data moves through your DuckDB SQL: which Parquet files and source tables feed each derived table, and through which joins, casts, filters, and aggregations. Gudu SQLFlow builds that map automatically: it parses your DuckDB scripts with a dedicated DuckDB dialect parser and renders an interactive lineage diagram, without executing a single query or touching your data.

Why DuckDB pipelines deserve real lineage

DuckDB has quietly become serious data infrastructure. It runs the transformation layer in dbt-duckdb projects, powers local analytics that later graduate to the warehouse, sits embedded inside applications and notebooks, and increasingly serves as the compute engine over Parquet data lakes. The SQL in those pipelines makes the same kinds of decisions warehouse SQL does: it computes revenue, filters customers, builds the tables a dashboard reads.

Yet DuckDB projects usually get none of the lineage rigor a warehouse gets. There is a structural reason: most lineage tooling assumes a database server. Runtime log-based lineage harvests query logs from a central engine; catalog-first platforms crawl a server’s metadata. DuckDB is in-process: there is often no server, no shared query history, and no catalog agent to install. The transformation logic lives in .sql files, dbt models, and scripts checked into a repo.

That makes static SQL analysis the natural, and often the only, way to build DuckDB data lineage. SQLFlow parses the SQL text itself, so it works the same whether that SQL runs on a laptop, in a CI job, or inside an application process.

How SQLFlow builds DuckDB data lineage

SQLFlow is built on the General SQL Parser (GSP), a commercial SQL compiler front-end developed since the mid-2000s and validated against roughly 13,600 per-dialect SQL test fixtures. DuckDB is one of 39 databases with a dialect-specific parser — not a generic ANSI grammar with DuckDB’s syntax bolted on. The pipeline:

1. Ingest. Paste DuckDB SQL, upload your script files, or import a dbt manifest from a dbt-duckdb project.
2. Parse and resolve. GSP builds a full semantic model of each statement, resolving every column reference through CTEs, subqueries, views, and SELECT * expansion.
3. Extract dataflow. The data-flow analyzer records, for each output column, exactly which source columns feed it and through which functions, casts, joins, and set operators.
4. Visualize and export. Drill through the interactive diagram, trace any column upstream or downstream, and export the graph as JSON, CSV, or PNG, or pull it through the REST API.

Example: lineage through read_parquet() and CTAS

The signature DuckDB pattern is CREATE TABLE AS SELECT reading files directly — no staging load, the Parquet files are the source. A typical reporting build:

CREATE TABLE reporting.daily_revenue AS
SELECT
    CAST(o.order_ts AS DATE)      AS order_date,
    c.region                      AS region,
    SUM(o.amount)                 AS revenue,
    COUNT(DISTINCT o.customer_id) AS buyers
FROM read_parquet('lake/orders/*.parquet') AS o
JOIN dim_customers AS c
  ON o.customer_id = c.customer_id
WHERE o.status = 'completed'
GROUP BY 1, 2;

SQLFlow’s DuckDB parser understands read_parquet() as a table source, so the Parquet path shows up as a real node in the lineage graph rather than an unresolved function call. From this one statement it extracts:

• daily_revenue.revenue is fed by o.amount from the Parquet source, through SUM().
• daily_revenue.order_date is fed by o.order_ts, through a CAST to DATE.
• daily_revenue.region comes straight from dim_customers.region.
• o.status, o.customer_id, and c.customer_id never land in the output, but they shape it, so SQLFlow records them as indirect lineage (more on that below).

Chain fifty of these statements across a dbt project or a nightly script and the graph compounds: SQLFlow stitches every intermediate table together so you can trace a dashboard number back to the exact Parquet column it started as.

Direct lineage vs indirect lineage

In the example above, o.status never appears in daily_revenue, yet changing how status is populated would silently change every revenue figure. SQLFlow models this as indirect (impact) lineage: columns used in WHERE, JOIN, and GROUP BY clauses and inside aggregates form a distinct, separately toggleable relationship type alongside direct dataflow. Most competing tools do not make this distinction, which means their impact analysis misses exactly the dependencies that cause silent data bugs.

For a DuckDB pipeline this matters twice over: file-based sources have no foreign keys or constraints to warn you, so the SQL itself is the only place these dependencies are recorded at all.

Lineage for dbt-duckdb projects

dbt-duckdb is one of the most common ways DuckDB runs in production, and dbt’s own DAG stops at model level: it tells you daily_revenue depends on stg_orders, not which columns carry the dependency. SQLFlow imports your dbt manifest and produces column-level lineage across the compiled models, so you can answer “which marts does stg_orders.amount actually feed?” before you change it.

The same analysis pays off at migration time. Teams frequently prototype on DuckDB and later promote models to ClickHouse, PostgreSQL, or a cloud warehouse. Because SQLFlow parses all of these with dialect-specific grammars, you can map the true dependency graph before the move and verify nothing was orphaned after it — using one tool and one lineage model on both sides.

What are the options for DuckDB lineage?

Approach	Good at	The gap for DuckDB
Manual documentation	Capturing intent and business context	Stale the day after it’s written; no column detail
Open-source parsers (sqllineage, sqlglot)	Parsing individual queries in a Python workflow; free	You assemble resolution, star-expansion, cross-statement stitching, and visualization yourself; indirect lineage is on you
Runtime log-based lineage	Observing what actually executed on a server	Assumes a central engine emitting logs; an in-process DuckDB on a laptop or inside an app has nothing to harvest
Catalog-first platforms	Governance workflows, ownership, glossaries across the whole stack	Lineage depth depends on their per-dialect SQL parsing; embedded DuckDB scripts are rarely first-class sources
Gudu SQLFlow	Static, column-level analysis of the SQL text itself, dialect-aware, with direct and indirect lineage	Commercial for team-scale use (free tier for individual queries)

These aren’t mutually exclusive. If you already run a catalog, SQLFlow’s enterprise deployments export lineage to DataHub, Microsoft Purview, and OpenMetadata, so it becomes the parsing engine behind the catalog you keep.

Deployment and privacy

SQLFlow performs static analysis of SQL code only; it never reads the rows in your tables or Parquet files. For teams whose DuckDB scripts encode proprietary logic, SQLFlow On-Premise runs as Docker or Kubernetes inside your network (air-gapped installs supported), so even the SQL text never leaves your infrastructure. SQLFlow Cloud starts free with premium at $49.99/month; On-Premise is $500/month or $4,800 one-time per selected database type, installable on two servers. Both are also scriptable through a headless CLI and REST API, which fits DuckDB’s automation-heavy, run-it-in-CI culture.

At the other end of the scale, the same engine batch-scans estates of 100+ databases and over a million columns with incremental scanning and a persistent lineage repository, so the DuckDB corner of your stack and the warehouse can live in one lineage graph.

Frequently asked questions

Can SQLFlow trace lineage through read_parquet() and file sources?

Yes. The DuckDB dialect parser treats read_parquet() as a table source, so file paths appear as source nodes in the lineage graph and columns read from Parquet are traced into downstream tables like any other column.

Do I have to run my DuckDB queries to get lineage?

No. SQLFlow is static analysis: it parses the SQL text and never executes it. That is what makes it a fit for in-process DuckDB, where there is no server-side query log to collect.

Does SQLFlow support dbt-duckdb projects?

Yes. Import the dbt manifest and SQLFlow produces column-level lineage across your models — finer-grained than dbt’s own model-level DAG.

Is DuckDB support a real dialect parser or generic ANSI SQL?

A real dialect parser. SQLFlow ships dialect-specific parsers for 39 databases, DuckDB among them, built on a parser engine validated against roughly 13,600 per-dialect SQL test fixtures.

Can lineage from DuckDB scripts feed my data catalog?

Yes. Enterprise deployments include export adapters for DataHub, Microsoft Purview, and OpenMetadata, plus JSON and CSV export and a REST API for custom integrations.

What does SQLFlow cost for DuckDB lineage?

SQLFlow Cloud has a free tier; premium is $49.99/month. On-Premise is $500/month or $4,800 one-time per selected database type. See pricing for details.