duckle 0.5.7

Local-first ETL/ELT pipelines that run on DuckDB. Headless CLI.

Duckle

Local-first ETL/ELT pipelines that run on DuckDB. Python builds the plan, DuckDB moves the rows. Your data never leaves the machine.

pip install duckle

That installs a ~20 MB native binary plus the DuckDB CLI. No JVM, no Docker, no server, no account.

---

No data passes through Python

Every method appends a node to a pipeline graph. Nothing executes until .run(), and then the whole graph is handed to the engine, compiled to SQL, and executed inside DuckDB.

That is the difference from a dataframe library: a billion-row job costs what the SQL costs, because it is SQL. There is no interpreter in the data path, and no to_pandas() escape hatch quietly pulling rows into memory.

import duckle
from duckle import col

(duckle.read_csv("orders.csv")
    .where(col.amount >= 20)
    .derive(total="round(amount * 1.2, 2)", tag="f'{region}-{id}'")
    .write_parquet("out.parquet")
    .run())

Python expressions, compiled to SQL

amount * 1.2, f'{a}-{b}', x if c else y, name.strip().upper() and email is None are translated to vectorized DuckDB SQL at plan time:

.derive(band="'high' if amount > 50 else 'low'")
# CASE WHEN ("amount" > 50) THEN 'high' ELSE 'low' END

An expression with no exact SQL equivalent is rejected with the construct named, never quietly executed somewhere slower. Comprehensions, lambda, indexing and eval are refused by name.

Prefer editor completion over strings? col builds the same tree with real operators, and fragments are reusable:

eu_only = col.region.isin(["EU", "UK"])
p.where(eu_only & (col.amount >= 20))

See exactly what will run, before it runs:

p.explain()   # prints the compiled SQL, one block per stage

359 components, not 10 file formats

Component ids map onto attribute paths, so anything in the catalog is reachable:

duckle.src.salesforce(object="Account", authMode="clientCredentials")
duckle.xf.geo.reproject(geomColumn="geom", targetCrs="EPSG:3857")
duckle.snk.salesforce.bulk(operation="upsert", externalIdField="Ext__c")

104 sources, 66 sinks and 138 transforms: Postgres, MySQL, Oracle, SQL Server, Snowflake, Databricks, Teradata, SAP OData, Salesforce (including Bulk API 2.0), Kafka, WebSocket, S3, SFTP, IMAP, LanceDB and more.

duckle.component_ids(contains="salesforce")
duckle.describe("snk.salesforce.bulk")   # settings, and which ones the engine ignores

A CI gate that needs no engine

duckle validate compiles every pipeline without opening a source or writing a sink, so it needs no DuckDB, no credentials and no network. Exit codes are stable and safe to gate on:

code	meaning
0	clean
1	a real finding: a pipeline failed, or did not compile
2	the runner could not start: bad usage, missing engine

duckle validate --json          # machine-readable, for a build step
duckle --pipeline my.json       # run one

Note: validate does not yet catch every missing required property, so a clean validate is not proof that a run will succeed.

Code and canvas are the same file

Pipelines are the same JSON the Duckle desktop studio reads. A pipeline written in Python opens on the canvas, and one drawn on the canvas runs from Python:

p.save("pipelines/orders.json")          # opens in the studio
duckle.from_json("pipelines/orders.json").run()

Install notes

duckle depends on duckdb-cli, published by the DuckDB Foundation, so the engine arrives with the install and works offline. To pin your own build instead, set DUCKLE_DUCKDB_BIN.

Wheels ship for Linux, macOS and Windows on x86-64 and arm64. The wheel carries a compiled Rust binary and is Python-version independent (py3-none-<platform>).

---

Apache-2.0  ·  GitHub  ·  duckle.org  ·  Issues