flowfile-duckdb 0.1.0

Query a Flowfile catalog from DuckDB / Superset — zero-copy, read-only.

flowfile-duckdb

Query a Flowfile catalog from DuckDB (and anything that speaks DuckDB, like Apache Superset) — zero-copy, read-only.

Flowfile stores catalog tables as open Delta Lake tables (Parquet + _delta_log) and tracks them in a small SQLite database (flowfile_catalog.db). DuckDB reads Delta natively via delta_scan, so this package is a thin metadata bridge: it reads the SQLite catalog and exposes every table to DuckDB as a view over delta_scan(...) (or read_parquet(...) for legacy Parquet tables). No data is copied or moved; queries read the underlying files live.

Install

With uv (recommended):

uv pip install flowfile-duckdb          # once published
uv pip install -e .                     # from a checkout

Or plain pip:

pip install flowfile-duckdb

CLI

flowfile-duckdb list                          # sanity-check discovered tables
flowfile-duckdb build --out catalog.duckdb    # persist a DuckDB file with the views
flowfile-duckdb attach > init.sql             # emit the CREATE VIEW SQL as an init script

Common flags: --catalog-db PATH, --schema-template, --include-cloud, --owner-id N.

Python

import duckdb
from flowfile_duckdb import attach_flowfile_catalog

con = duckdb.connect()
attach_flowfile_catalog(con)                  # schemas render as e.g. Demo_sales_analytics
con.sql('SELECT * FROM "Demo_sales_analytics"."sales" LIMIT 5')

Schema names default to {catalog}_{schema} (single underscore); override via schema_template= (e.g. "{schema}" to drop the catalog prefix).

Where is the catalog DB?

Resolved the same way Flowfile resolves it:

1. FLOWFILE_DB_PATH — explicit path to flowfile_catalog.db.
2. <FLOWFILE_STORAGE_DIR or ~/.flowfile>/database/flowfile_catalog.db (default).

The SQLite connection is opened read-only.

Apache Superset

pip install duckdb-engine
flowfile-duckdb build --out /data/flowfile_catalog.duckdb

Add a Superset database with SQLAlchemy URI duckdb:////data/flowfile_catalog.duckdb. See examples/superset.md.

Publishing (maintainers)

Distribution is automated via GitHub Actions (.github/workflows/publish.yml), using PyPI Trusted Publishing (OIDC — no API token stored):

1. On PyPI, add a trusted publisher for this project: owner edwardvaneechoud, repo flowfile-duckdb, workflow publish.yml, environment pypi.
2. Create a GitHub Environment named pypi.
3. Push a v* tag (or publish a GitHub Release) — the workflow builds with uv and publishes to PyPI.

git tag v0.1.0 && git push origin v0.1.0

Known limitations / next steps

• Cloud tables (s3://, az://, gs://) are skipped unless --include-cloud, which also needs a DuckDB CREATE SECRET. flowfile_duckdb/secrets.py maps a decrypted options dict to CREATE SECRET SQL but deliberately does not decrypt Flowfile's $ffsec$ secrets (that needs Flowfile's master key + HKDF).
• Virtual / SQL tables have no physical files and are skipped (they can't be read without Flowfile's engine).
• Refresh: views read live data, but the table list is a snapshot — re-run build/attach to pick up newly created tables.
• On multi-user (docker-mode) installs the catalog holds every user's tables; use --owner-id to scope to one user.

License

MIT