Disclaimer: This is a personal project. It is not affiliated with, endorsed by, or supported by any employer or vendor. No warranty — use it at your own risk.
duckrun runs SQL in DuckDB and reads/writes Delta Lake via delta-rs — locally or on OneLake / S3 / GCS / ADLS. It's just glue: DuckDB executes · delta-rs materializes · Arrow bridges · dbt orchestrates. Two ways to use it:
connect()— a notebook helper to query and write Delta straight from SQL (this page);- a dbt adapter that materializes models as Delta tables.
Concurrent writers are first-class: every write is snapshot-pinned and fails loud rather than silently interleaving.
Install
In a Microsoft Fabric notebook, upgrade and restart the kernel (duckrun needs duckdb ≥ 1.5.4,
which is newer than the bundled stable build; it fails loud at connect() otherwise):
!pip install duckrun --upgrade
notebookutils.session.restartPython()
Quickstart — OneLake in a notebook
import duckrun
# Read-only by default — explore a lakehouse safely, no chance of an accidental write.
# Use the workspace + lakehouse GUIDs (friendly names hit an upstream OneLake read bug for now).
conn = duckrun.connect("abfss://<workspace_id>@onelake.dfs.fabric.microsoft.com/<lakehouse_id>/Tables/dbo")
conn.sql("SHOW TABLES").show()
conn.sql("select status, count(*) from orders group by status").show()
df = conn.table("orders").toPandas() # or .toArrow() for a streaming RecordBatchReader
# Time travel: list the versions, then read one
from duckrun import DeltaTable
DeltaTable.forName(conn, "orders").history() # newest-first commits: version, timestamp, operation, …
conn.read.format("delta").option("versionAsOf", 0).load(".../Tables/dbo/orders").show()
Need to write? Opt in with read_only=False:
conn = duckrun.connect("abfss://…/Tables/dbo", read_only=False)
# write Delta straight from SQL
conn.sql("select * from orders where amount > 0") \
.write.mode("overwrite").saveAsTable("clean_orders")
# raw DML routes to delta-rs (insert / update / delete / create table as / alter / drop)
conn.sql("delete from clean_orders where amount = 0")
# upsert — snapshot-pinned automatically, nothing extra to pass
from duckrun import DeltaTable
src = conn.sql("select * from updates")
DeltaTable.forName(conn, "clean_orders").merge(src, "target.id = source.id") \
.whenMatchedUpdateAll().whenNotMatchedInsertAll().execute()
conn.stop()
Multiple catalogs — attach more lakehouses and read/join across them by three-part name. In
Fabric a Warehouse is just a write-locked Lakehouse, so attach it read_only=True next to a writable
one:
conn.attach("abfss://…/warehouse.Warehouse/Tables", name="warehouse", read_only=True)
conn.attach("/data/reference", name="local")
conn.sql("select * from warehouse.mart.facts f join local.dbo.lookup l on l.id = f.id").show()
Works the same against a local path, s3://, gs://, or az://. Full method map:
Connection API · Spark/Delta coverage ·
live multi-catalog demo.
dbt adapter
duckrun is also a dbt adapter — a thin wrapper around
dbt-duckdb that adds Delta-backed table / incremental
materializations (everything else dbt-duckdb gives you is inherited). Point a profile at a lakehouse
and dbt run:
# ~/.dbt/profiles.yml
my_project:
outputs:
dev:
type: duckrun
root_path: "abfss://<workspace_id>@onelake.dfs.fabric.microsoft.com/<lakehouse_id>/Tables"
Multiple lakehouses in one project — declare extra write roots as named catalogs: and send a
model to one with the standard dbt +database: <alias> config (e.g. a Bronze/Silver/Gold medallion
across three Fabric Lakehouses). ref() and joins resolve across them:
dev:
type: duckrun
root_path: "abfss://ws@onelake.dfs.fabric.microsoft.com/LH_Silver.Lakehouse/Tables" # default
catalogs:
lh_bronze: { root_path: "abfss://ws@onelake.dfs.fabric.microsoft.com/LH_Bronze.Lakehouse/Tables" }
lh_gold: { root_path: "abfss://ws@onelake.dfs.fabric.microsoft.com/LH_Gold.Lakehouse/Tables" }
-- models/bronze/raw_events.sql → lands in LH_Bronze
{{ config(materialized='incremental', database='lh_bronze', unique_key='id') }}
select ...
Profiles, materializations, incremental strategies (incl. append_if_unchanged), sources, and
automatic compaction/vacuum are all in docs/dbt-adapter.md.
See it on real projects: aemo and coffee are
runnable starters, and parity_tests/ runs real type: duckdb projects (jaffle_shop,
sde, MRR, TechFlow, Tuva) unchanged on duckrun and diffs the output against dbt-duckdb.
Building with an AI assistant
duckrun ships a guide so AI coding assistants get the adapter's defaults right (several differ from other dbt adapters). For Claude Code:
/plugin marketplace add djouallah/duckrun
/plugin install duckrun-projects@duckrun
Other assistants read the AGENTS.md at the repo root, which points to the full guide.
None of this is required to use duckrun.
How it works
Two engines, split cleanly: DuckDB runs every query and reads Delta through delta_scan views,
delta-rs handles every write, an Arrow C-stream bridges them, and dbt orchestrates on top.
Writes are snapshot-pinned: the read is fixed at delta_scan(…, version => N) and the write commits
against N, so a concurrent commit is rejected with CommitFailedError instead of silently
overwriting a lost update.
More on the design: Design document · Snapshot isolation.
Docs
Browse the rendered docs site at djouallah.github.io/duckrun — or read the markdown here:
| Doc | What's in it |
|---|---|
| Connection API | The duckrun.connect() notebook API + the live per-method scorecard. |
| Spark / Delta coverage | What the connect() surface maps to in PySpark / Delta. |
| dbt adapter | Profiles, materializations, incremental strategies, sources, maintenance, limitations. |
| Design document | Why delta-rs (not DuckDB's native Delta writer), why Delta (not Iceberg), why a separate adapter. |
| Snapshot isolation | How a read-modify-write is fenced to the version you read, and how it compares to delta-rs/Spark/SQL Server. |
| dbt adapter conformance | Official dbt-tests-adapter results, regenerated on every push to main. |
| Incremental MERGE benchmark | ~60M-row TPCH merge / append / overwrite scorecard — the release gate. |
License
MIT
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file duckrun-0.3.34.tar.gz.
File metadata
- Download URL: duckrun-0.3.34.tar.gz
- Upload date:
- Size: 139.8 kB
- Tags: Source
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/6.1.0 CPython/3.13.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
663fb31e9d9823caeead2a238fa868f4e82572cc193a8fce0ac20e2a7eb0aa14
|
|
| MD5 |
e4a3206c47ea3790aa419106f931db9b
|
|
| BLAKE2b-256 |
76d2f230aec32cd713adb482c58a692323bd92b563b384a83c054a8f67a34b76
|
Provenance
The following attestation bundles were made for duckrun-0.3.34.tar.gz:
Publisher:
publish.yml on djouallah/duckrun
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
duckrun-0.3.34.tar.gz -
Subject digest:
663fb31e9d9823caeead2a238fa868f4e82572cc193a8fce0ac20e2a7eb0aa14 - Sigstore transparency entry: 2068277039
- Sigstore integration time:
-
Permalink:
djouallah/duckrun@e48c8f084f92c2aa1d904bc4890eb8066018e865 -
Branch / Tag:
refs/tags/v0.3.34 - Owner: https://github.com/djouallah
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
publish.yml@e48c8f084f92c2aa1d904bc4890eb8066018e865 -
Trigger Event:
push
-
Statement type:
File details
Details for the file duckrun-0.3.34-py3-none-any.whl.
File metadata
- Download URL: duckrun-0.3.34-py3-none-any.whl
- Upload date:
- Size: 148.3 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? Yes
- Uploaded via: twine/6.1.0 CPython/3.13.12
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
1538c7f8bfc99cf213edcff4fe7cc40a3b5d148d079da1b0b64f79f228989723
|
|
| MD5 |
07847f3ec0ccac14dd73961579d05b78
|
|
| BLAKE2b-256 |
9b39549da062c1f005b55f9a062940a694c1e1c11d8169718b520ea2177d6f53
|
Provenance
The following attestation bundles were made for duckrun-0.3.34-py3-none-any.whl:
Publisher:
publish.yml on djouallah/duckrun
-
Statement:
-
Statement type:
https://in-toto.io/Statement/v1 -
Predicate type:
https://docs.pypi.org/attestations/publish/v1 -
Subject name:
duckrun-0.3.34-py3-none-any.whl -
Subject digest:
1538c7f8bfc99cf213edcff4fe7cc40a3b5d148d079da1b0b64f79f228989723 - Sigstore transparency entry: 2068277255
- Sigstore integration time:
-
Permalink:
djouallah/duckrun@e48c8f084f92c2aa1d904bc4890eb8066018e865 -
Branch / Tag:
refs/tags/v0.3.34 - Owner: https://github.com/djouallah
-
Access:
public
-
Token Issuer:
https://token.actions.githubusercontent.com -
Runner Environment:
github-hosted -
Publication workflow:
publish.yml@e48c8f084f92c2aa1d904bc4890eb8066018e865 -
Trigger Event:
push
-
Statement type:
