SQLFlow Ingester (formerly Grabit) collects SQL from your data estate — the DDL of tables, views, stored procedures, functions, triggers, and packages — and feeds it to SQLFlow for column-level data lineage analysis. It reads from live databases over JDBC, from revision systems like GitHub and Bitbucket, and from the local file system. It extracts metadata only and never reads the rows in your tables.
Run it in 3 minutes: point the ingester at a database, and it writes a metadata bundle SQLFlow turns into an interactive column-level lineage diagram. Everything runs from the command line, so it schedules cleanly as a cron job.
Collect SQL from three kinds of source
Connect to a database
Extract DDL — CREATE TABLE, CREATE VIEW, stored procedures, functions, triggers, and constraints — from Oracle, SQL Server, Snowflake, PostgreSQL, and 9 more dialects over JDBC, and send it to SQLFlow for analysis.
Connect to a revision system
Fetch SQL files straight from GitHub, Bitbucket, and other revision systems, and send them to SQLFlow to build the data lineage from the code your team already version-controls.
Search the file system
Find SQL files in a local directory tree and push them to SQLFlow to visualize the data lineage — handy for scripts and exports that never made it into a database or a repo.
Two export formats: single JSON and sharded
When the ingester reads a database, it writes a normalized metadata bundle — catalog structure plus the DDL text of every view, procedure, function, trigger, and foreign key. One model covers every supported dialect; each vendor simply omits the object types it doesn’t have. That bundle comes in two shapes.
| Single JSON | Sharded (default) | |
|---|---|---|
| On disk | One metadata.json file | A directory: manifest.json + catalog/ + source/ + index/ |
| Where DDL lives | Inline, in servers[].queries[] | In source/*.jsonl, located by a byte-offset index/*.idx |
| Best for | Small and medium catalogs | Large estates — catalog stays resident, multi-GB DDL streams and seeks |
| Extras | Simplest to consume | Per-shard SHA-256, per-object content hashing, incremental exports, block compression |
Single JSON puts the whole estate in one file with every object’s DDL inline — the simplest thing to parse. Sharded is the default for large estates: it streams the export with bounded memory, keeps the structural catalog small and resident, makes the DDL random-seekable through an index, and writes its manifest last so a consumer never sees a half-written export. Full field-by-field references: single JSON export and sharded export.
Discover data lineage in a SQL Server database in 3 minutes
Run it from the command line, on a schedule
The ingester is a self-contained Java tool you run from the command line, so it schedules cleanly as a cron job for automated, repeatable extraction. Passwords can be supplied from an environment variable or a file so they never appear in the process arguments, and fine-grained include/exclude filters control exactly which databases, schemas, and objects are extracted. See the SQLFlow Ingester documentation to get started.
Get lineage from your database estate
Download the SQLFlow Ingester, point it at a database, and turn its metadata into column-level data lineage.