Skip to main content

interloper-duckdb

DuckDB tables as an interloper destination: a DuckDBDestination writing to a local .duckdb file or a MotherDuck database, and the DuckDBConnection that opens it.

Setup

A local file needs nothing but a path; DuckDB creates the file on first write:

from interloper_duckdb import DuckDBConnection

connection = DuckDBConnection(database="./warehouse.duckdb")

A MotherDuck database is named md:<database> and authenticates with a service token from the MotherDuck settings page:

connection = DuckDBConnection(database="md:my_db", motherduck_token="...")

Both fields also load from the environment (DUCKDB_DATABASE, DUCKDB_MOTHERDUCK_TOKEN), so DuckDBConnection() works with no arguments.

Usage

import interloper as il
from interloper_duckdb import DuckDBConnection, DuckDBDestination

destination = DuckDBDestination(
    connection=DuckDBConnection(database="./warehouse.duckdb"),
    default_dataset="raw",
)

In a deployed instance you configure this through the UI instead: add a DuckDB connection, then a DuckDB destination using it.

Datasets are schemas

An asset's dataset is a DuckDB schema. A table lands in the asset's dataset, else in the destination's default_dataset, else in main. The schema and the table are created on the first write, the columns typed from the asset's schema (or from one inferred from the data), and never altered afterwards: a column the table does not have is dropped from the write with a warning.

Field type Column type
bool BOOLEAN
int BIGINT
float DOUBLE
Decimal DECIMAL(38,9)
datetime TIMESTAMP
date DATE
bytes BLOB
str, Any VARCHAR
nested model, list[...] JSON

Partitions

A write replaces what it covers, in one transaction: the whole table for an unpartitioned asset, the rows inside a time partition's bounds (day >= start AND day < end), or the rows equal to a partition's id for any other partitioning. A window deletes each partition it covers and inserts the whole batch once. If the insert fails, the delete is rolled back and the table keeps its previous rows.

One writer per file

A local DuckDB file admits one writing process at a time. Concurrent assets in one process are fine (each write runs on its own cursor), but two processes writing to the same file, such as two pods or a scheduler and a notebook, fail to open it. A connection holds the file from its first use until its process exits, and that includes the connection check the app runs from the API process. Run the instance's writes in one process, or use MotherDuck, which serves many writers.

Querying the tables

The tables are plain DuckDB tables, so any DuckDB client reads them:

duckdb warehouse.duckdb -c 'SELECT * FROM raw.ads_stats LIMIT 10'
import duckdb

duckdb.connect("warehouse.duckdb", read_only=True).sql("SELECT * FROM raw.ads_stats").df()

Another process can open the file only while no process holds it for writing. Open it read_only, so the reader does not lock the instance out in turn.

Metadata

Release files for interloper-duckdb 0.122.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for interloper-duckdb 0.122.0
File Size Uploaded Provenance Actions
interloper_duckdb-0.122.0.tar.gz 8.9 kB Without provenance Details for interloper_duckdb-0.122.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for interloper-duckdb 0.122.0
File Interpreter ABI Platform Actions
interloper_duckdb-0.122.0-py3-none-any.whl Python 3 none any Details for interloper_duckdb-0.122.0-py3-none-any.whl

Total release size: 19.3 kB

Release files / interloper_duckdb-0.122.0.tar.gz

Download URL interloper_duckdb-0.122.0.tar.gz
Size 8.9 kB
Tags Source
SHA-256 checksum
How to use checksums
cc91f0586aa20677b745c7b080fe73f1af2e1b8f855a9b5646aad0f77c9fdc16
BLAKE2b-256 checksum
How to use checksums
2cde8b16a12d7cf3ed7a5add94084d4c2b2d2e62084adb5c7f06638fbdf0ea04
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.6 {"installer":{"name":"uv","version":"0.13.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Release files / interloper_duckdb-0.122.0-py3-none-any.whl

Download URL interloper_duckdb-0.122.0-py3-none-any.whl
Size 10.4 kB
Tags Python 3
SHA-256 checksum
How to use checksums
0c7c4511b7113dce84dcb1c4b187128df0e623aba654a8f3b72bb203a89a9d89
BLAKE2b-256 checksum
How to use checksums
d4b062ed29d353f3c3b8429146b7f9760c938b245cef549e7db516dbb3fe0e07
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via uv/0.13.6 {"installer":{"name":"uv","version":"0.13.6","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}

Security for interloper-duckdb 0.122.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.122.0 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page