Skip to main content

Deisa-Dask

Dask-Enabled In Situ Analytics for HPC simulations.

Deisa couples MPI-based scientific simulations with Dask distributed computing, enabling real-time, scalable analysis of simulation data as it is produced. Simulation data living on MPI ranks is streamed to Dask workers, where callbacks run analytics (reductions, plotting, machine learning, …) on the latest timestep without waiting for the simulation to finish.

This repository is the Dask integration layer of the Deisa project. It depends on deisa-core for the shared MPI-side protocol.

How it works

Deisa runs two cooperating sides:

  • Bridge — created inside the MPI simulation. Each rank builds a Bridge describing the arrays it owns (global shape, chunk shape, chunk position) and calls bridge.send(name, local_data, timestep=ts) each timestep to push its chunk to the Dask cluster.
  • Deisa — runs on the analytics side (a Dask client). Callbacks are registered with @deisa.register("array_name") and are executed on the Dask workers whenever a new timestep for that array arrives. deisa.execute_callbacks() blocks until the simulation finishes and all callbacks have run.

The two sides connect through a Dask scheduler; the scheduler address is communicated to the simulation via the DEISA_DASK_SCHEDULER_ADDRESS environment variable.

Key features

  • Per-array MPI communicators — each distributed array gets its own communicator (MPI.Comm.Dup() / MPI.Comm.Split()), so arrays with different decomposition patterns coexist cleanly.
  • Non-distributed arrays — arrays that live on a single rank are handled natively.
  • Round-robin worker scatter — successive timesteps are scattered across workers so data is not piled onto a single worker.
  • Worker list control — update_worker / filter_worker let a Bridge adjust which workers receive data before a send().
  • MPI test harness — the test suite runs both with a FakeComm mock (fast, pytest-xdist-parallel) and against real OpenMPI / MPICH.

Installation

Requires Python ≥ 3.10.

# core
pip install deisa-dask

# with MPI support (for running simulations / MPI tests)
pip install "deisa-dask[mpi]"

# development / test dependencies
pip install "deisa-dask[test]"

Getting started

A complete runnable example lives in example/getting-started/. It runs a 4-rank MPI simulation that pushes a 2-D Gaussian field each timestep, and an analytics process that sums and plots the latest field.

Simulation side (simulation.py) — create a Bridge, describe your arrays, and send each timestep:

from mpi4py import MPI
from deisa.dask import Bridge

comm = MPI.COMM_WORLD
rank = comm.Get_rank()

bridge = Bridge(
    comm=comm,
    arrays_metadata={
        "temperature": {
            "global_shape": (64, 64),
            "chunk_shape": (32, 32),
            "chunk_position": rank_coords,   # this rank's position in the cartesian grid
        }
    },
)

for ts in range(5):
    local_data = compute_local_field(ts)     # this rank's chunk
    bridge.send("temperature", local_data, timestep=ts)

bridge.close(timestep=5)

Analytics side (analysis.py) — register a callback and block until done:

from deisa.dask import Deisa

deisa = Deisa()

@deisa.register("temperature")
def callback(temperatures):
    latest = temperatures[-1]
    print(f"t={latest.timestep} sum={latest.sum().compute()}")

deisa.execute_callbacks()

Launch (launch.sh) — start a Dask scheduler and worker, then the analytics and simulation processes. The scheduler address is exported to the simulation through DEISA_DASK_SCHEDULER_ADDRESS.

export DEISA_DASK_SCHEDULER_ADDRESS="tcp://localhost:8786"
mpirun -np 1 dask scheduler --scheduler-file scheduler.json &
mpirun -np 1 dask worker --scheduler-file scheduler.json &
mpirun -np 1 python3 analysis.py &
mpirun -np 4 python3 simulation.py

Documentation

API documentation is published from the site/ Hugo project. See the documentation website for the full reference.

Development

git clone https://github.com/deisa-project/deisa-dask.git
cd deisa-dask
uv sync --extra test --extra mpi   # or: pip install -e ".[test,mpi]"

# run the non-MPI suite (FakeComm mock, parallel via pytest-xdist)
pytest -n auto

# run the MPI suite against a real MPI implementation
pytest test/test_mpi.py

Code style is enforced with Ruff (line length 120). The test suite covers the FakeComm harness, per-array communicators, handshake, and MPI integration; CI runs MPI tests against both OpenMPI and MPICH and tracks time-to-callback benchmarks.

Changelog

See CHANGELOG.md (Keep a Changelog, Semantic Versioning).

License

MIT © deisa-project.

Metadata

Release files for deisa-dask 0.7.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for deisa-dask 0.7.0
File Size Uploaded Provenance Actions
deisa_dask-0.7.0.tar.gz 199.8 kB With provenance Details for deisa_dask-0.7.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for deisa-dask 0.7.0
File Interpreter ABI Platform Actions
deisa_dask-0.7.0-py3-none-any.whl Python 3 none any Details for deisa_dask-0.7.0-py3-none-any.whl

Total release size: 319.0 kB

Release files / deisa_dask-0.7.0.tar.gz

Download URL deisa_dask-0.7.0.tar.gz
Size 199.8 kB
Tags Source
SHA-256 checksum
How to use checksums
87743c0a9e33c0b2cb6251dac12e3b879406a2ba0f0613ee0c1f48bdad02792f
BLAKE2b-256 checksum
How to use checksums
334c153ed9ea1cfc8330a7667704e78217bd81f803402edd06f476e229e79a70
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 9, 2026.

Transparency log

Release files / deisa_dask-0.7.0-py3-none-any.whl

Download URL deisa_dask-0.7.0-py3-none-any.whl
Size 119.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
3e8297a9e43156ee5a8811da759eb876c64ca2f53c2b7d8e9d217cfc4a8d8a71
BLAKE2b-256 checksum
How to use checksums
ed039a15732c4b67eddbe4625ff285b2ebd37487eb09b3fdd4d05f5203d9b111
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 9, 2026.

Transparency log

Security for deisa-dask 0.7.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for deisa_dask-0.7.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 9, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page