Skip to main content

Duckstring

Get your ducks in a row.

Duckstring is an open source data engineering platform built on DuckDB. Data into the terabytes fits comfortably on a single machine, which is where DuckDB is at its best, and Duckstring provides what a pipeline needs around it: packaging and versioning, orchestration, incremental processing, a catalog and delivery. It runs the same on a laptop, on a single server, or with work spread across cloud machines.

Documentation · Quickstart · Playground

Versioned transformations

Each set of transformations is a Pond: a Python project whose pond.toml names it, gives it a semantic version and lists the Ponds it reads from.

[pond]
name = "sales"
version = "1.2.0"

[sources]
transactions = "1.0.0"
products = "1.1.0"

The pipeline follows from those declarations, so it never has to be drawn by hand. A breaking change ships as a new major version, which runs alongside the old one while downstream Ponds move across at their own pace.

Pull orchestration

Schedules are set on the Ponds whose output is actually used, at the end of the pipeline. Duckstring works back from there, running each upstream Pond only as often as something downstream needs it, so a path nobody consumes never runs. A daily job is one command:

duckstring refresh add reports --every 1d

Data can also stay fresh just because it's being used: a Tap gives an app its own key, and its queries refresh the Pond when the data is older than you allow, so nothing runs while the app is idle.

The Playground runs the demo pipeline in your browser, with a short guided tour of how it's scheduled.

Incremental processing

A Trickle stores a table's changes as a Z-set: rows weighted +1 when added and -1 when removed. Joins and aggregations written with the Trickle builder recompute only the keys that changed since the last run, and handle updates and deletes correctly across any shape of join:

from duckstring import ripple


@ripple
def priced_line(pond):
    (
        pond.trickle("orders.order_line")
        .join(pond.trickle("catalog.product"), on="product_id")
        .select(
            "s0.order_id, s0.product_id, s0.quantity, s1.unit_price, "
            "CAST(round(s0.quantity * s1.unit_price, 2) AS DECIMAL(14,2)) AS revenue"
        )
        .merge("priced_line", pk="order_id")
    )

Code and data together

The runtime, called the Catchment, is also the catalog. Each major version of a Pond is a schema holding its tables, with lineage, run history and the version's schema contract recorded alongside. Query it from the CLI, from Python, or from any Postgres client:

SELECT * FROM reports_v1.monthly_summary;

Python and the command line

Ripples, the steps inside a Pond, are Python functions with a DuckDB connection, so SQL, DuckDB's relation API, pandas and Arrow all work. Everything a Catchment does can be driven from the duckstring CLI or its HTTP API, and a web UI shows pipelines as they run.

Quickstart

Duckstring needs Python 3.10 or newer.

pip install duckstring
duckstring catchment init --name dev

This starts a local Catchment, with its web UI at http://127.0.0.1:7474. Leave it running, and in a second terminal:

mkdir demo && cd demo
duckstring pond demo                       # transactions, products → sales → reports
duckstring pond deploy --all --yes
duckstring refresh now reports                 # run the pipeline once, from the end
duckstring query reports.monthly_summary

The Quickstart goes through the same steps with explanations.

Also included

  • Local testing against small samples of each Source, called Puddles
  • dbt projects deployed as Ponds, one step per model
  • Apps: dashboards and small web apps, such as Streamlit or FastAPI, served by the Catchment beside the data they read and started when someone opens them
  • Delivery to Postgres (incremental and exactly once), a DuckLake, S3, Azure Storage, Google Cloud Storage and local files
  • Databricks: read Unity Catalog tables and Volume files into Ponds, deliver back to Unity Catalog, and send large recomputes to a SQL warehouse
  • Retries, alerts and Prometheus metrics
  • Per-Pond Python environments from pyproject.toml and uv.lock
  • Cloud compute on AWS, Azure or Google Cloud, chosen per Pond, with data on S3, Azure Storage or Google Cloud Storage
  • Connected Catchments, where a Pond reads from a Pond on another Catchment

Documentation

docs.duckstring.com has the concepts, task guides, and a reference for the CLI, Python API, pond.toml and HTTP API. To try pull orchestration without installing anything, the Playground simulates it in the browser, with a guided tour.

Questions and discussion

Ask questions, share what you're building or suggest ideas in GitHub Discussions. Bugs go in Issues.

Status

Duckstring is pre-1.0, so interfaces may still change between minor versions. Changes are listed in the changelog.

Hosted Catchments

A hosted Catchment service is planned at duckstring.com. If you're interested, get in touch at dev@duckstring.com.

License

Apache 2.0

Metadata

Release files for duckstring 0.8.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for duckstring 0.8.0
File Size Uploaded Provenance Actions
duckstring-0.8.0.tar.gz 924.2 kB With provenance Details for duckstring-0.8.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for duckstring 0.8.0
File Interpreter ABI Platform Actions
duckstring-0.8.0-py3-none-any.whl Python 3 none any Details for duckstring-0.8.0-py3-none-any.whl

Total release size: 2.0 MB

Release files / duckstring-0.8.0.tar.gz

Download URL duckstring-0.8.0.tar.gz
Size 924.2 kB
Tags Source
SHA-256 checksum
How to use checksums
c642542b571efe3cc5e0076de02c30a6afdbd9d8f08d6095ae981681267dea11
BLAKE2b-256 checksum
How to use checksums
bafce5ed8e9e1fdc947b9bbd2f64d2ad49b48d300bbde7b6570f1d6424960551
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 11, 2026.

Transparency log

Release files / duckstring-0.8.0-py3-none-any.whl

Download URL duckstring-0.8.0-py3-none-any.whl
Size 1.0 MB
Tags Python 3
SHA-256 checksum
How to use checksums
728984284fb05ae0fa62b2600abce0a57585a7d2f26a088ce3e061ed77731291
BLAKE2b-256 checksum
How to use checksums
5e7a87df952a65f798aa2bc26207eec2ee23c839f37c15dd868729d232a648b1
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 11, 2026.

Transparency log

Security for duckstring 0.8.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for duckstring-0.8.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 11, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page