Greenplum Data Lineage: Column Lineage for MPP Warehouses

Traçabilité des données Greenplum is the column-level map of how data flows through a Greenplum warehouse: from external tables and staging schemas, through INSERT-SELECT ELT chains and views, into the distributed fact and dimension tables that reports read. Gudu SQLFlow builds that map automatically by parsing your Greenplum SQL with a dedicated Greenplum dialect parser — one of 39 dialect-specific parsers it ships — and rendering an interactive diagram you can trace column by column.

Why Greenplum data lineage needs a dedicated parser

Greenplum is a PostgreSQL-family MPP database, and most of its query syntax will look familiar to anyone who knows Postgres. But the parts of a Greenplum estate that carry the most lineage weight are exactly the parts that diverge from vanilla PostgreSQL:

• External tables. CREATE EXTERNAL TABLE with gpfdist or file locations is the standard front door for data landing in Greenplum. A parser that chokes on the LOCATION et FORMAT clauses loses the first hop of every load pipeline.
• Distribution keys. DISTRIBUTED BY et DISTRIBUTED RANDOMLY clauses appear in nearly every table definition. Generic ANSI grammars reject them, which means they reject your DDL, which means no lineage at all for those tables.
• INSERT-SELECT ELT chains. Greenplum warehouses typically transform data in-database: staging table to conformed table to fact table, each step an INSERT INTO ... SELECT with joins, casts, and aggregates. Lineage has to be stitched across every hop of the chain, not computed per statement in isolation.

SQLFlow handles all three because its Greenplum support is a distinct dialect parser, not PostgreSQL with the errors suppressed. The engine underneath is the Analyseur SQL général, a commercial SQL compiler front-end developed since the mid-2000s and validated against roughly 13,600 per-dialect SQL test fixtures.

Worked example: external table load into a distributed fact table

Here is a compact version of the pattern most Greenplum load pipelines follow. Daily order files arrive via gpfdist, land in an external table, and are merged into a fact table distributed on id_de_commande:

CREATE EXTERNAL TABLE ext_stage.daily_orders (
    order_id     bigint,
    customer_id  bigint,
    order_ts     timestamp,
    amount       numeric(12,2),
    channel      text
)
LOCATION ('gpfdist://etl-host:8081/daily_orders*.csv')
FORMAT 'CSV' (HEADER);

CREATE TABLE dw.fact_orders (
    order_id     bigint,
    customer_sk  bigint,
    order_date   date,
    net_amount   numeric(12,2)
)
DISTRIBUTED BY (order_id);

INSERT INTO dw.fact_orders (order_id, customer_sk, order_date, net_amount)
SELECT o.order_id,
       c.customer_sk,
       o.order_ts::date,
       o.amount - COALESCE(r.refund_amount, 0)
FROM   ext_stage.daily_orders o
JOIN   dw.dim_customer  c ON c.customer_id = o.customer_id
LEFT JOIN dw.stg_refunds r ON r.order_id  = o.order_id
WHERE  o.channel <> 'test';

Run this through SQLFlow and the diagram shows, for each column of dw.fact_orders, exactly where the data came from and what happened to it on the way:

Colonne cible	Source columns	Transformation
fact_orders.order_id	ext_stage.daily_orders.order_id	Direct copy
fact_orders.customer_sk	dw.dim_customer.customer_sk	Direct, via join on identifiant_client
fact_orders.order_date	ext_stage.daily_orders.order_ts	Cast to date
fact_orders.net_amount	daily_orders.amount, stg_refunds.refund_amount	Subtraction with COALESCE

Note what a table-level view would have hidden: net_amount is fed by two different tables, and the external table’s channel column never reaches the fact table at all — yet it still controls which rows do. SQLFlow captures that second kind of relationship too.

Direct lineage vs indirect lineage in ELT chains

In the example above, channel appears only in the OÙ clause and the join keys appear only in SUR conditions. None of them lands in the output, but changing any of them changes the numbers in fact_orders. SQLFlow models these as lignée indirecte (d'impact), a separate relationship type from direct dataflow, and lets you toggle each independently in the diagram. Most lineage tools do not make this distinction, which means their impact analysis silently misses filter and join dependencies.

This matters more in Greenplum than in most environments because of how deep the INSERT-SELECT chains run. A typical estate stacks external table, staging table, conformed layer, fact layer, and reporting views on top of each other. SQLFlow resolves column references through every hop, including CTEs, subqueries, views, and SÉLECTIONNER * expansion, so a trace from a report column walks back to the source file feed in one continuous path.

Migrating off Greenplum? Map dependencies first

Many Greenplum estates are now candidates for migration to cloud warehouses, and the single biggest source of migration overruns is unknown dependencies: the view nobody documented, the staging table three consumers quietly read, the column a downstream report needs that the new model dropped. A dependency map built before you move is how you avoid discovering these in production.

A lineage-first migration workflow with SQLFlow looks like this:

1. Scan the estate. Feed SQLFlow your Greenplum DDL, view definitions, and load scripts — pasted, uploaded as files, or pulled live over JDBC. Enterprise deployments batch-scan estates of 100+ databases and more than a million columns, with incremental rescans as the code changes.
2. Find what is actually used. Column-level lineage separates the tables and columns that feed live outputs from the ones nothing reads, so you migrate the real warehouse instead of twenty years of accretion.
3. Sequence the move. The lineage graph gives you the dependency order: which subject areas can move independently and which drag a chain of upstream feeds with them.
4. Verify the target. Because the same engine parses the destination dialect — Snowflake, BigQuery, Redshift, Databricks, and the rest of the 39 supported dialects — you can diagram the migrated SQL and compare lineage before and after.

Teams running mixed legacy estates use the same approach on other MPP platforms: see the companion pages on Traçabilité des données Teradata et Vertica data lineage. And because Greenplum is PostgreSQL-derived, estates that also run vanilla Postgres can cover both with one tool — the Traçabilité des données PostgreSQL page covers the differences.

How to get Greenplum SQL into SQLFlow

• Paste or upload: individual queries, scripts, or batches of .sql files in the browser.
• JDBC metadata: connect to Greenplum and pull table definitions and view SQL directly, so lineage reflects what is deployed rather than what is in the repo.
• Grabit ingester: automated metadata extraction for scheduled, repeatable scans.
• REST API and CLI: drive analysis from CI or an orchestration pipeline via the API REST SQLFlow or the headless CLI, and export results as JSON, CSV, or PNG.

Everything is static analysis of SQL text and schema metadata. SQLFlow never reads the rows in your tables, and the On-Premise edition (Docker or Kubernetes, air-gap friendly) keeps even the SQL text inside your network — relevant for the regulated industries where Greenplum is common. Lineage can also be exported to DataHub, Microsoft Purview, or OpenMetadata if one of those catalogs is your system of record.

What about open-source lineage tools?

Les analyseurs syntaxiques open source comme SQL Lineage et sqlglot are genuinely useful for extracting table relationships from individual, well-behaved queries, and if that is your whole problem they may be enough. The gap shows up on production Greenplum code: CREATE EXTERNAL TABLE et DISTRIBUTED BY syntax, multi-hop INSERT-SELECT chains that need cross-statement stitching, SÉLECTIONNER * expansion that requires schema metadata, and indirect lineage through filters and joins. Run one of your real load scripts through both and compare the output — that test settles the question faster than any feature matrix.

Foire aux questions

Does SQLFlow support Greenplum-specific syntax like external tables and DISTRIBUTED BY?

Yes. SQLFlow ships a dedicated Greenplum dialect parser, so CREATE EXTERNAL TABLE, DISTRIBUTED BY clauses, and other Greenplum DDL parse cleanly and participate in the lineage graph rather than causing errors.

Can I just use the PostgreSQL dialect for Greenplum?

Plain SELECT statements would mostly parse, but the statements that define your load pipelines — external table DDL and distributed table definitions — are Greenplum-specific. Use the Greenplum dialect so the whole estate parses; SQLFlow supports both, so mixed Greenplum-and-Postgres shops are covered.

Does SQLFlow need access to the data in my Greenplum tables?

No. SQLFlow performs static analysis of SQL code and optionally reads schema metadata over JDBC. It never reads table rows, and the On-Premise edition keeps SQL text entirely inside your network.

Can SQLFlow help plan a Greenplum-to-cloud migration?

Yes. Scan the Greenplum estate to get the true column-level dependency graph, use it to scope and sequence the migration, then diagram the migrated SQL on the target platform — Snowflake, BigQuery, Redshift, Databricks, and 35 other dialects are supported by the same engine — to verify nothing was orphaned.

How much does SQLFlow cost for Greenplum lineage?

SQLFlow Cloud has a free tier; premium is $49.99/month. SQLFlow On-Premise is $500/month or $4,800 one-time per selected database type, installable on two servers. See tarification pour plus de détails.