daisy

Blockwise task scheduler for processing large volumetric data.

Daisy v2 is a Rust rewrite of the original daisy, with a focus on faster scheduling, deterministic block dependencies, and resumable runs via on-disk done markers. The Python API is built on a Rust core (daisy-core) exposed through PyO3 (daisy._daisy).

Upgrading from daisy 1.x? v2 is a deliberate breaking release. Several 1.x APIs have moved or been removed (daisy.persistence, daisy.Array, daisy.open_ds, daisy.prepare_ds). See MIGRATION.md for what changed and how to move over, and CHANGELOG.md for the full list. If you depend on the 1.x surface, pin daisy<2.

Branches were renamed for 2.0

With the 2.0.0 release (2026-10-21) the branches changed:

If you install from PyPI, nothing changes. pip install daisy gets 2.x; pin daisy<2 to stay on 1.x.

If you install from git, references to the old branch names no longer resolve. Replace them:

• git+https://github.com/funkelab/daisy@master → daisy<2 from PyPI, or @v1
• git+https://github.com/funkelab/daisy@v2.0 → daisy>=2 from PyPI, or @main

If you have a local clone, rename your branch and point it at the new remote branch:

git fetch origin --prune
git branch -m master v1  && git branch -u origin/v1 v1      # if you had master
git branch -m v2.0 main  && git branch -u origin/main main  # if you had v2.0
git remote set-head origin -a

Install

pip install daisy

Requires Python ≥ 3.11. Wheels for Linux (glibc + musl, x86_64/aarch64), macOS (x86_64/arm64), and Windows are built in CI (.github/workflows/publish.yaml) and published to PyPI on version tags; building from source needs a Rust toolchain.

Quick example

import daisy


def process(block):
    # do work on block.read_roi, write to block.write_roi
    print(f"processing {block.block_id}: {block.write_roi}")


task = daisy.Task(
    task_id="example",
    total_roi=daisy.Roi((0,), (1000,)),
    read_roi=daisy.Roi((0,), (10,)),
    write_roi=daisy.Roi((0,), (10,)),
    process_function=process,
    read_write_conflict=False,
    max_workers=4,
)

daisy.run_blockwise([task])

For larger end-to-end examples (mutex watershed, multi-task pipelines, JSON progress observers, stress tests), see examples/.

Concepts

A task declares a total_roi, a read_roi, and a write_roi. Daisy tiles the total ROI into blocks at the stride of the write ROI, and dispatches blocks subject to the dependency constraints implied by overlap between read and write ROIs (within a task and across upstream/downstream tasks).

A skip mask (Task(skip_mask=…)) is an array with one cell per block, nonzero meaning "skip this block this run"; a tracking path (Task(tracking_path=…)) persists which blocks completed so a rerun resumes. The two compose: done blocks stay done, masked blocks are reported as skipped but never recorded as done.

A block function receives one block at a time. A worker function takes zero arguments — daisy spawns one per worker slot, and the worker drives its own daisy.Client loop. Worker-function mode is the right choice when you have per-worker setup (model load, DB connection, mmap'd array) you want to amortise across many blocks.

Documentation

Full documentation (including executable tutorials and the design notes) is built with Sphinx + MyST-NB and published to GitHub Pages. To build locally:

uv sync --group docs
uv run sphinx-build -b html docs/source docs/_build/html

The design notes are also browsable directly under docs/source/design/:

• ARCHITECTURE.md — system overview
• SCHEDULER.md — dependency graph and ready surface
• PROTOCOL.md — TCP wire format
• DONE_MARKERS.md — on-disk persistence for resumable runs
• RUN_STATS.md — per-worker and per-task statistics
• WORKER_POOL_COORDINATION.md — resource budgets
• WORKER_SHUTDOWN_FLOWS.md — shutdown sequencing
• ABANDONMENT.md — typestate model for task lifecycle

License

MIT — see LICENSE.