Traçabilité des données Hive : Traçabilité au niveau des colonnes pour HQL et Hadoop ETL

Hive data lineage is the column-level map of how data flows through your HQL: which source tables and columns feed each Hive table, partition, and view, and through which joins, aggregations, and LATERAL VIEW expansions. The most reliable way to build it is to parse the HQL itself — Gudu SQLFlow ships a dedicated Hive dialect parser that turns your scripts into an interactive, column-level lineage diagram without touching your cluster.

Why Hive data lineage is harder than it looks

A typical Hadoop warehouse accumulates years of HQL: staging loads, partitioned INSERT OVERWRITE jobs, views over external tables, and Oozie- or Airflow-scheduled scripts nobody has read since the person who wrote them left. Answering “what feeds dw.revenu_quotidien?” or “what breaks if we drop staging.orders.status?” means tracing column references through all of it.

Generic SQL parsers stumble on the constructs that make HiveQL its own dialect:

• LATERAL VIEW explode() — one input column of arrays or maps fans out into multiple rows and new derived columns. Lineage has to connect the exploded output columns back to the single source collection.
• Partitioned INSERT OVERWRITE — the partition column may come from a static literal in the PARTITION clause or dynamically from the last SELECT columns. Both cases change which columns count as lineage sources.
• External tables — the table is a schema over files in HDFS or object storage. Lineage must treat it as a first-class source even though no upstream SQL ever wrote it.
• Hive-specific syntax — DISTRIBUTE BY, CLUSTER BY, multi-insert (FROM ... INSERT ... INSERT ...), SerDe-backed types, and backtick-quoted identifiers all break ANSI-only grammars.

SQLFlow parses HiveQL with a dedicated dialect parser, one of 39 dialect-specific parsers in the tool — a grammar built for HiveQL, not a generic ANSI parser with exceptions bolted on. The underlying engine, the Analyseur SQL général, has been developed commercially since the mid-2000s and is validated against roughly 13,600 per-dialect SQL test fixtures.

Atlas hooks vs. parsing HQL: two ways to get Hive lineage

Most Hadoop teams first meet lineage through Apache Atlas. Atlas is genuinely good at what it was built for: its Hive hook sits on the cluster and captures lineage as jobs actually execute, giving you a live governance record of what ran, when, tied into classification and tagging across the Hadoop stack.

The hook approach has structural limits, though, and they matter most exactly when teams go looking for lineage:

	Runtime hooks (Atlas-style)	Parsing HQL (SQLFlow)
Needs a running cluster	Yes — lineage is captured only when jobs execute with the hook installed	No — static analysis of the SQL text, anywhere
Covers code that hasn’t run	No — a script that never executed leaves no trace	Yes — every script in the repo, including dead or rarely-run jobs
Works during/after migration	Hard — once the cluster is decommissioned, so is the capture point	Yes — analyze the exported HQL offline, before, during, and after the move
History coverage	Starts when you installed the hook	Complete — the code is the record
Reads your data	Runs inside the cluster alongside jobs	Never — SQL text and schema metadata only

These approaches are complementary rather than rivals: hooks tell you what ran; a parser tells you what the code does. If your question is “document every dependency in this Hive estate so we can audit it or move it”, parsing is the tool that answers it — and SQLFlow can push its results into DataHub, Microsoft Purview, or OpenMetadata if one of those is your system of record.

Column-level lineage from a real HiveQL job

Here is the shape of job every Hadoop warehouse runs nightly — a partitioned INSERT OVERWRITE built from joined staging tables:

INSERT OVERWRITE TABLE dw.daily_revenue PARTITION (ds = '2026-07-11')
SELECT c.region,
       SUM(o.amount)              AS revenue,
       COUNT(DISTINCT o.order_id) AS order_cnt
FROM staging.orders o
JOIN staging.customers c
  ON o.customer_id = c.customer_id
WHERE o.ds = '2026-07-11'
  AND o.status = 'COMPLETE'
GROUP BY c.region;

Run this through SQLFlow and the diagram shows, per output column:

• dw.daily_revenue.revenue is fed directly by mise en place.commandes.montant à travers SOMME().
• dw.daily_revenue.order_cnt est alimenté par staging.orders.order_id à travers COUNT(DISTINCT).
• dw.daily_revenue.region maps straight from staging.customers.region.
• The static partition value populates dw.daily_revenue.ds — SQLFlow models the partition column as a lineage target even though it never appears in the SELECT list.
• staging.orders.status, staging.orders.ds, and both identifiant_client columns shape the result through the OÙ et REJOINDRE conditions without landing in the output. SQLFlow records these as lignée indirecte (d'impact), a separate, toggleable relationship type. Most lineage tools do not make this distinction, yet it is exactly what impact analysis needs: change statut coding and every revenue number downstream moves.

The same column resolution works through LATERAL VIEW explode(): if a downstream table selects item.sku from an exploded order_events.items array, SQLFlow traces référence back to the source collection column, through the table alias the lateral view introduced. It also resolves references through CTEs, nested subqueries, views, and SÉLECTIONNER * expansion, so star-heavy legacy HQL still yields precise column edges.

Hive lineage at migration time

The single most common reason teams need Hive lineage today is that they are leaving Hive. Whether the destination is Spark, Databricks, or a cloud warehouse, the questions are the same: which jobs feed which tables, in what order do they have to move, and which of these 4,000 scripts are actually dead?

Because SQLFlow works from the code alone, you can run the analysis on an export of your HQL repository from any machine — no access to the (possibly already frozen) cluster required. Batch scanning is built for this: SQLFlow scans estates of 100+ databases and over a million columns, keeps results in a persistent lineage repository, and refreshes incrementally as scripts change during the migration. And since Hive, Spark SQL, et Databricks each have their own dialect parser in the same tool, you can analyze the source estate and the rewritten target estate side by side and verify the new lineage matches the old.

How to run it

• Paste or upload HQL in SQLFlow Cloud — free tier, results in the browser, exportable as JSON, CSV, or PNG.
• Pull metadata from the metastore over JDBC so view definitions and schemas resolve SÉLECTIONNER * and cross-script references correctly.
• Automate it through the API REST or headless CLI — for example, analyze every HQL change in CI before it merges.
• Keep everything in-network with SQLFlow sur site: Docker or Kubernetes, air-gapped if needed, $500/month or $4,800 one-time per selected database type. Many Hadoop estates sit in banks and telcos where SQL text cannot leave the network; this edition exists for them.

In every deployment, SQLFlow performs static analysis only: it reads SQL code and schema metadata, never the rows in your tables. Since version 8.2.3 you can also query the resulting graph in plain English (“which tables depend on staging.orders?”), with every table and column in the answer validated against the analyzed graph before display. For the broader capability tour, see the Présentation de l'outil de traçabilité des données SQL.

Foire aux questions

SQLFlow peut-il construire la lignée Hive sans accès au cluster ?

Yes. SQLFlow parses HQL text, so an export of your scripts (plus DDL for accurate SÉLECTIONNER * resolution) is enough. That is the key difference from hook-based approaches like Apache Atlas, which capture lineage only when jobs execute on a live cluster with the hook installed.

Gère-t-il la commande LATERAL VIEW explode et d'autres syntaxes spécifiques à Hive ?

Yes. Hive has its own dedicated parser among SQLFlow’s 39 dialect parsers, covering LATERAL VIEW explode(), partitioned INSERT OVERWRITE, and external tables.

La lignée est-elle au niveau des colonnes ou au niveau de la table ?

Column-level. For each output column, SQLFlow identifies the exact source columns that feed it and the functions, casts, joins, and set operators along the way. It additionally records indirect lineage — columns used in OÙ, REJOINDRE, et GROUPER PAR clauses that shape the result without appearing in it.

Puis-je exporter la lignée Hive vers mon catalogue de données ?

Oui. Les déploiements en entreprise incluent des adaptateurs d'exportation pour DataHub, Microsoft Purview et OpenMetadata, ainsi que l'exportation JSON et CSV et une API REST pour les intégrations personnalisées.

Est-ce que SQLFlow lit les données stockées dans mes tables Hive ?

No. It performs static analysis of SQL code and optionally reads schema metadata from the metastore. Table row data is never touched, and with the On-Premise edition the SQL text itself never leaves your network.

Combien coûte SQLFlow ?

SQLFlow Cloud starts free; premium accounts are $49.99/month. SQLFlow On-Premise is $500/month or $4,800 one-time per selected database type, installable on two servers, with additional database types at $100/month or $1,000 one-time each.