Redshift Data Lineage: Automated Column Lineage from Query Logs

Redshift data lineage is the column-level map of how data moves through your Amazon Redshift cluster: which source tables and columns feed each target table, view, and report, and which transformations happen along the way. Redshift keeps no such map itself — but it does log every SQL statement it executes, and in Redshift every transformation is expressed as SQL. Parse those logs and you can reconstruct complete lineage automatically. That is exactly what Gudu SQLFlow does: it ingests Redshift query logs natively and turns the executed SQL into interactive, column-level lineage diagrams.

Why Redshift has no built-in lineage view

The Redshift console and system views are built for operating the cluster, not for understanding data flow. You can inspect query performance, queue behavior, and table statistics, but nothing in Redshift links an output column back to the source columns that produced it. Even the PostgreSQL-style catalog dependency tracking that Redshift inherited breaks down for one of Redshift’s own features: late-binding views (created WITH NO SCHEMA BINDING) deliberately record no dependency on their underlying tables.

What Redshift does keep is the raw material: system tables such as STL_QUERYTEXT e SYS_QUERY_HISTORY store the text of executed statements, and audit logging can archive every query to S3 for long-term retention. The SQL history is all there. What is missing is the analysis layer that reads thousands of those statements and answers “where does fact_orders.net_revenue actually come from?” That analysis layer is a SQL lineage tool.

How to get Redshift data lineage from query logs

Query-log-based lineage has one decisive advantage over analyzing only your source repositories: it captures what actually ran. Scheduled ETL, ad hoc backfills, statements issued by BI tools and orchestrators, one-off fixes someone ran from a SQL client — all of it lands in the query history, whether or not it lives in version control. The workflow with SQLFlow:

1. Collect executed SQL. Pull statement text from Redshift’s system tables, or from the audit logs Redshift writes to S3. Redshift query log ingestion is a native SQLFlow input, alongside pasted SQL, uploaded files, and live metadata over JDBC.
2. Parse with a Redshift-specific parser. SQLFlow analyzes each statement with a dedicated Redshift dialect parser — one of 39 dialect-specific parsers, not a generic ANSI grammar — and resolves every column reference through CTEs, subqueries, views, and SELECT * expansion.
3. Merge and explore. Lineage from every statement is merged into one graph you can drill into interactively, trace upstream or downstream from any column, and export as JSON, CSV, or PNG, or query over the SQLFlow REST API.

Because this is static analysis of SQL text, SQLFlow never reads the rows in your tables. It only needs the SQL and, optionally, schema metadata to resolve ambiguous references.

A worked example: INSERT … SELECT with a window function

Here is the kind of statement that fills a real Redshift query log — an aggregation with a window function loading a reporting table:

INSERT INTO analytics.customer_monthly_rank
    (customer_id, order_month, monthly_revenue, revenue_rank)
SELECT
    o.customer_id,
    DATE_TRUNC('month', o.order_date)       AS order_month,
    SUM(o.amount)                           AS monthly_revenue,
    RANK() OVER (
        PARTITION BY DATE_TRUNC('month', o.order_date)
        ORDER BY SUM(o.amount) DESC
    )                                       AS revenue_rank
FROM sales.orders o
WHERE o.status = 'complete'
GROUP BY o.customer_id, DATE_TRUNC('month', o.order_date);

From this single statement, SQLFlow extracts column-level lineage in two distinct categories:

Target column	Direct sources	Indirect (impact) sources
customer_id	orders.customer_id	orders.status (WHERE)
order_month	orders.order_date via DATE_TRUNC	orders.status
monthly_revenue	orders.amount via SUM	orders.status, orders.customer_id, orders.order_date (GROUP BY)
revenue_rank	orders.amount via SUM then RANK() OVER	orders.order_date (PARTITION BY), orders.status

Note the second category. orders.status never lands in the output, yet changing how it is populated changes every number in the target table. SQLFlow models this indirect lineage — columns acting through WHERE, GROUP BY, JOIN conditions, and window partitions — as a separate, toggleable relationship type. Most lineage tools do not make this distinction, and it is precisely the difference between “which reports read this column” and “which reports are affected by this column.”

Redshift-specific constructs are still just SQL

Late-binding views

Late-binding views are the standard Redshift pattern for decoupling views from underlying tables, and they are invisible to catalog-based dependency queries by design. To a SQL parser, though, a late-binding view is simply a view definition: SQLFlow parses the view’s SELECT and connects its output columns to their sources like any other view, so lineage flows straight through WITH NO SCHEMA BINDING.

DISTKEY, SORTKEY, and DISTSTYLE

Distribution and sort keys shape performance, not data flow. A table declared with DISTKEY(customer_id) SORTKEY(order_date) carries the same lineage as any other table; SQLFlow parses the Redshift DDL, records the table and its columns, and the physical tuning clauses pass through without affecting the graph. Your lineage stays correct whether a table is DISTSTYLE ALL, EVEN, ou KEY.

Redshift is not PostgreSQL

Redshift descends from PostgreSQL but has diverged substantially in syntax and behavior. That is why SQLFlow ships a dedicated Redshift parser rather than reusing its PostgreSQL lineage parser — each of the 39 supported dialects gets its own grammar, validated against a corpus of roughly 13,600 per-dialect SQL test fixtures built up over two decades of commercial parser development.

Ways to feed Redshift SQL into SQLFlow

Input	What it gives you
Redshift query logs	Lineage from every executed statement — the complete, ground-truth picture
Live metadata over JDBC	DDL and view definitions pulled directly from the cluster
Uploaded SQL files	ETL scripts and repositories analyzed as a batch
Pasted SQL	Instant lineage for a single statement in the browser
dbt manifest	Column-level lineage across dbt models that build in Redshift

At enterprise scale, SQLFlow batch-scans estates of 100+ databases and over a million columns, runs incremental scans, maintains a persistent lineage repository, and exports to DataHub, Microsoft Purview, and OpenMetadata — so Redshift lineage can feed the catalog you already run rather than living in another silo.

How does this compare to other approaches?

Open-source parsers such as sqllineage e sqlglot are solid for extracting lineage from individual, well-formed statements, and for a handful of clean queries they may be enough. The gap opens on a production Redshift log: thousands of statements, Redshift-specific syntax, views layered on views, SELECT * that needs schema metadata to expand, and indirect lineage through filters and window partitions. Catalog-first platforms are strong at organizing and governing metadata across many systems; for deep SQL analysis they typically need a specialized lineage engine underneath — which is why SQLFlow ships export adapters for DataHub, Purview, and OpenMetadata instead of competing with them.

If you run Redshift alongside other warehouses, the same engine covers them with the same approach — see Snowflake data lineage from query history, which works much like Redshift’s log-based ingestion. And for regulated environments, SQLFlow no local runs in Docker or Kubernetes inside your network, so your SQL text never leaves your infrastructure.

Frequently asked questions

Does SQLFlow read the data in my Redshift cluster?

No. SQLFlow performs static analysis of SQL code and optionally reads schema metadata (table and column definitions). It never reads table rows. With the On-Premise edition, even the SQL text stays inside your network.

Where does the Redshift SQL come from?

From wherever your executed statements live: Redshift’s query history system tables, audit logs archived to S3, your ETL script repositories, or DDL and view definitions pulled live over JDBC. Redshift query log ingestion is a native SQLFlow input, so lineage reflects what actually ran on the cluster.

Are late-binding views a problem for lineage?

They defeat catalog-based dependency tracking, because Redshift records no dependency for them by design. They are not a problem for parser-based lineage: SQLFlow analyzes the view’s SQL definition directly and connects its columns to their sources like any other view.

Does SQLFlow show which columns only influence results through filters?

Yes. SQLFlow distinguishes direct lineage (data actually flowing into an output column) from indirect lineage (columns acting through WHERE, JOIN, GROUP BY, and window partition clauses), and lets you toggle each independently in the diagram.

How much does SQLFlow cost?

SQLFlow Cloud starts free; premium accounts are $49.99/month. SQLFlow On-Premise is $500/month or $4,800 one-time per selected database type, installable on two servers, with each additional database type at $100/month or $1,000 one-time.