Skip to main content
Pre-release

This release is a pre-release and may not be stable for production use.

Duckstring

Get your ducks in a row.

Duckstring is an open source data engineering platform built on DuckDB. Data into the terabytes fits comfortably on a single machine, which is where DuckDB is at its best, and Duckstring provides what a pipeline needs around it: packaging and versioning, orchestration, incremental processing, a catalog and delivery. It runs the same on a laptop, on a single server, or with work spread across cloud machines.

Documentation · Quickstart · Playground

Versioned transformations

Each set of transformations is a Pond: a Python project whose pond.toml names it, gives it a semantic version and lists the Ponds it reads from.

[pond]
name = "sales"
version = "1.2.0"

[sources]
transactions = "1.0.0"
products = "1.1.0"

The pipeline follows from those declarations, so it never has to be drawn by hand. A breaking change ships as a new major version, which runs alongside the old one while downstream Ponds move across at their own pace.

Pull orchestration

Schedules are set on the Ponds whose output is actually used, at the end of the pipeline. Duckstring works back from there, running each upstream Pond only as often as something downstream needs it, so a path nobody consumes never runs. A daily job is one command:

duckstring refresh add reports --every 1d

Data can also stay fresh just because it's being used: a Tap gives an app its own key, and its queries refresh the Pond when the data is older than you allow, so nothing runs while the app is idle.

The Playground runs the demo pipeline in your browser, with a short guided tour of how it's scheduled.

Incremental processing

A Trickle stores a table's changes as a Z-set: rows weighted +1 when added and -1 when removed. Joins and aggregations written with the Trickle builder recompute only the keys that changed since the last run, and handle updates and deletes correctly across any shape of join:

from duckstring import ripple


@ripple
def priced_line(pond):
    (
        pond.trickle("orders.order_line")
        .join(pond.trickle("catalog.product"), on="product_id")
        .select(
            "s0.order_id, s0.product_id, s0.quantity, s1.unit_price, "
            "CAST(round(s0.quantity * s1.unit_price, 2) AS DECIMAL(14,2)) AS revenue"
        )
        .merge("priced_line", pk="order_id")
    )

Code and data together

The runtime, called the Catchment, is also the catalog. Each major version of a Pond is a schema holding its tables, with lineage, run history and the version's schema contract recorded alongside. Query it from the CLI, from Python, or from any Postgres client:

SELECT * FROM reports_v1.monthly_summary;

Python and the command line

Ripples, the steps inside a Pond, are Python functions with a DuckDB connection, so SQL, DuckDB's relation API, pandas and Arrow all work. Everything a Catchment does can be driven from the duckstring CLI or its HTTP API, and a web UI shows pipelines as they run.

Quickstart

Duckstring needs Python 3.10 or newer.

pip install duckstring
duckstring catchment init --name dev

This starts a local Catchment, with its web UI at http://127.0.0.1:7474. Leave it running, and in a second terminal:

mkdir demo && cd demo
duckstring pond demo                       # transactions, products → sales → reports
duckstring pond deploy --all --yes
duckstring refresh now reports                 # run the pipeline once, from the end
duckstring query reports.monthly_summary

The Quickstart goes through the same steps with explanations.

Also included

Documentation

docs.duckstring.com has the concepts, task guides, and a reference for the CLI, Python API, pond.toml and HTTP API. To try pull orchestration without installing anything, the Playground simulates it in the browser, with a guided tour.

Status

Duckstring is pre-1.0, so interfaces may still change between minor versions. Changes are listed in the changelog.

Hosted Catchments

A hosted Catchment service is planned at duckstring.com. If you're interested, get in touch at dev@duckstring.com.

License

Apache 2.0

Metadata

Release files for duckstring 0.7.1rc2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for duckstring 0.7.1rc2
File Size Uploaded Provenance Actions
duckstring-0.7.1rc2.tar.gz 801.7 kB With provenance Details for duckstring-0.7.1rc2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for duckstring 0.7.1rc2
File Interpreter ABI Platform Actions
duckstring-0.7.1rc2-py3-none-any.whl Python 3 none any Details for duckstring-0.7.1rc2-py3-none-any.whl

Total release size: 1.7 MB

Release files / duckstring-0.7.1rc2.tar.gz

Download URL duckstring-0.7.1rc2.tar.gz
Size 801.7 kB
Tags Source
SHA-256 checksum
How to use checksums
ed2859219023fa51c3cc5d7cbfbf620d4cdacff0985f95d96634528275ceb230
BLAKE2b-256 checksum
How to use checksums
e12ed226ac8da38551509b2e191ea8b4bca916618a49c64161d552cad4eff900
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 27, 2026.

Transparency log

Release files / duckstring-0.7.1rc2-py3-none-any.whl

Download URL duckstring-0.7.1rc2-py3-none-any.whl
Size 906.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
338b9fccebb554f7532db527132250277b561feae016eeacd3e8e692d866538b
BLAKE2b-256 checksum
How to use checksums
0b25ea7cd5ffa126c9c97bfe5b911ffbafa752be3045b0de7447d5fd826ac02b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 27, 2026.

Transparency log

Security for duckstring 0.7.1rc2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for duckstring-0.7.1rc2.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 27, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page