interloper-duckdb
DuckDB tables as an interloper destination: a DuckDBDestination writing to a
local .duckdb file or a MotherDuck database, and the DuckDBConnection that
opens it.
Setup
A local file needs nothing but a path; DuckDB creates the file on first write:
from interloper_duckdb import DuckDBConnection
connection = DuckDBConnection(database="./warehouse.duckdb")
A MotherDuck database is named md:<database> and authenticates with a
service token from the MotherDuck settings page:
connection = DuckDBConnection(database="md:my_db", motherduck_token="...")
Both fields also load from the environment (DUCKDB_DATABASE,
DUCKDB_MOTHERDUCK_TOKEN), so DuckDBConnection() works with no arguments.
Usage
import interloper as il
from interloper_duckdb import DuckDBConnection, DuckDBDestination
destination = DuckDBDestination(
connection=DuckDBConnection(database="./warehouse.duckdb"),
default_dataset="raw",
)
In a deployed instance you configure this through the UI instead: add a DuckDB connection, then a DuckDB destination using it.
Datasets are schemas
An asset's dataset is a DuckDB schema. A table lands in the asset's dataset,
else in the destination's default_dataset, else in main. The schema and
the table are created on the first write, the columns typed from the asset's
schema (or from one inferred from the data), and never altered afterwards: a
column the table does not have is dropped from the write with a warning.
| Field type | Column type |
|---|---|
bool |
BOOLEAN |
int |
BIGINT |
float |
DOUBLE |
Decimal |
DECIMAL(38,9) |
datetime |
TIMESTAMP |
date |
DATE |
bytes |
BLOB |
str, Any |
VARCHAR |
nested model, list[...] |
JSON |
Partitions
A write replaces what it covers, in one transaction: the whole table for an
unpartitioned asset, the rows inside a time partition's bounds
(day >= start AND day < end), or the rows equal to a partition's id for any
other partitioning. A window deletes each partition it covers and inserts the
whole batch once. If the insert fails, the delete is rolled back and the table
keeps its previous rows.
One writer per file
A local DuckDB file admits one writing process at a time. Concurrent assets in one process are fine (each write runs on its own cursor), but two processes writing to the same file, such as two pods or a scheduler and a notebook, fail to open it. A connection holds the file from its first use until its process exits, and that includes the connection check the app runs from the API process. Run the instance's writes in one process, or use MotherDuck, which serves many writers.
Querying the tables
The tables are plain DuckDB tables, so any DuckDB client reads them:
duckdb warehouse.duckdb -c 'SELECT * FROM raw.ads_stats LIMIT 10'
import duckdb
duckdb.connect("warehouse.duckdb", read_only=True).sql("SELECT * FROM raw.ads_stats").df()
Another process can open the file only while no process holds it for
writing. Open it read_only, so the reader does not lock the instance out in
turn.
Metadata
Release files for interloper-duckdb 0.109.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| interloper_duckdb-0.109.0.tar.gz | 7.9 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| interloper_duckdb-0.109.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 17.8 kB
Release files / interloper_duckdb-0.109.0.tar.gz
| Download URL | interloper_duckdb-0.109.0.tar.gz |
|---|---|
| Size | 7.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
9e7768d854c8e84d2181b389c36692d938c356abb8e80a6f4cdb23aed5cdcb1a
|
|
BLAKE2b-256 checksum How to use checksums |
ee6966f48f3806494d5aa77d4d61156c3b2c6ed5fa41cd144882302df1e21819
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.12.23 {"installer":{"name":"uv","version":"0.12.23","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / interloper_duckdb-0.109.0-py3-none-any.whl
| Download URL | interloper_duckdb-0.109.0-py3-none-any.whl |
|---|---|
| Size | 9.8 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
4a4572f5f0ba1e901461c334a2156d07d977d73408246cda801dfe999eafa73d
|
|
BLAKE2b-256 checksum How to use checksums |
e85c2fd5aed6fb21325ce4aa84c1d62acc4af0005e16d3f88b2b32c714ddbb36
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.12.23 {"installer":{"name":"uv","version":"0.12.23","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
