Deisa-Dask
Dask-Enabled In Situ Analytics for HPC simulations.
Deisa couples MPI-based scientific simulations with Dask distributed computing, enabling real-time, scalable analysis of simulation data as it is produced. Simulation data living on MPI ranks is streamed to Dask workers, where callbacks run analytics (reductions, plotting, machine learning, …) on the latest timestep without waiting for the simulation to finish.
This repository is the Dask integration layer of the Deisa project. It depends
on deisa-core for the shared
MPI-side protocol.
How it works
Deisa runs two cooperating sides:
Bridge— created inside the MPI simulation. Each rank builds aBridgedescribing the arrays it owns (global shape, chunk shape, chunk position) and callsbridge.send(name, local_data, timestep=ts)each timestep to push its chunk to the Dask cluster.Deisa— runs on the analytics side (a Dask client). Callbacks are registered with@deisa.register("array_name")and are executed on the Dask workers whenever a new timestep for that array arrives.deisa.execute_callbacks()blocks until the simulation finishes and all callbacks have run.
The two sides connect through a Dask scheduler; the scheduler address is
communicated to the simulation via the DEISA_DASK_SCHEDULER_ADDRESS
environment variable.
Key features
- Per-array MPI communicators — each distributed array gets its own
communicator (
MPI.Comm.Dup()/MPI.Comm.Split()), so arrays with different decomposition patterns coexist cleanly. - Non-distributed arrays — arrays that live on a single rank are handled natively.
- Round-robin worker scatter — successive timesteps are scattered across workers so data is not piled onto a single worker.
- Worker list control —
update_worker/filter_workerlet aBridgeadjust which workers receive data before asend(). - MPI test harness — the test suite runs both with a
FakeCommmock (fast,pytest-xdist-parallel) and against real OpenMPI / MPICH.
Installation
Requires Python ≥ 3.10.
# core
pip install deisa-dask
# with MPI support (for running simulations / MPI tests)
pip install "deisa-dask[mpi]"
# development / test dependencies
pip install "deisa-dask[test]"
Getting started
A complete runnable example lives in example/getting-started/.
It runs a 4-rank MPI simulation that pushes a 2-D Gaussian field each timestep,
and an analytics process that sums and plots the latest field.
Simulation side (simulation.py) — create a Bridge, describe your arrays,
and send each timestep:
from mpi4py import MPI
from deisa.dask import Bridge
comm = MPI.COMM_WORLD
rank = comm.Get_rank()
bridge = Bridge(
comm=comm,
arrays_metadata={
"temperature": {
"global_shape": (64, 64),
"chunk_shape": (32, 32),
"chunk_position": rank_coords, # this rank's position in the cartesian grid
}
},
)
for ts in range(5):
local_data = compute_local_field(ts) # this rank's chunk
bridge.send("temperature", local_data, timestep=ts)
bridge.close(timestep=5)
Analytics side (analysis.py) — register a callback and block until done:
from deisa.dask import Deisa
deisa = Deisa()
@deisa.register("temperature")
def callback(temperatures):
latest = temperatures[-1]
print(f"t={latest.timestep} sum={latest.sum().compute()}")
deisa.execute_callbacks()
Launch (launch.sh) — start a Dask scheduler and worker, then the analytics
and simulation processes. The scheduler address is exported to the simulation
through DEISA_DASK_SCHEDULER_ADDRESS.
export DEISA_DASK_SCHEDULER_ADDRESS="tcp://localhost:8786"
mpirun -np 1 dask scheduler --scheduler-file scheduler.json &
mpirun -np 1 dask worker --scheduler-file scheduler.json &
mpirun -np 1 python3 analysis.py &
mpirun -np 4 python3 simulation.py
Documentation
API documentation is published from the site/ Hugo project. See the
documentation website for the
full reference.
Development
git clone https://github.com/deisa-project/deisa-dask.git
cd deisa-dask
uv sync --extra test --extra mpi # or: pip install -e ".[test,mpi]"
# run the non-MPI suite (FakeComm mock, parallel via pytest-xdist)
pytest -n auto
# run the MPI suite against a real MPI implementation
pytest test/test_mpi.py
Code style is enforced with Ruff (line length 120). The test suite covers the FakeComm harness, per-array communicators, handshake, and MPI integration; CI runs MPI tests against both OpenMPI and MPICH and tracks time-to-callback benchmarks.
Changelog
See CHANGELOG.md (Keep a Changelog, Semantic Versioning).
License
MIT © deisa-project.
Metadata
Release files for deisa-dask 0.7.1
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| deisa_dask-0.7.1.tar.gz | 215.2 kB | With provenance | Details for deisa_dask-0.7.1.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| deisa_dask-0.7.1-py3-none-any.whl | Python 3 | none | any | Details for deisa_dask-0.7.1-py3-none-any.whl |
Total release size: 337.5 kB
Release files / deisa_dask-0.7.1.tar.gz
| Download URL | deisa_dask-0.7.1.tar.gz |
|---|---|
| Size | 215.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
3bd09971cea5e58c745a4101e81c943e7e3167806835ddafe8d6a4072d50f336
|
|
BLAKE2b-256 checksum How to use checksums |
676d96161ad94caeb5849a7decf3d431b2539324a0e0e8257f30ad1ef383a8bd
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.
Transparency logRelease files / deisa_dask-0.7.1-py3-none-any.whl
| Download URL | deisa_dask-0.7.1-py3-none-any.whl |
|---|---|
| Size | 122.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
f19872828c58f0fd4419ece72624b53336e7f3b7eae59f9de6fd4c2c90112a00
|
|
BLAKE2b-256 checksum How to use checksums |
c39c8822c3b161969218aa8f3d31406969e4f0df07818b7d2ce51c53f5d301ab
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.
Transparency logSecurity for deisa-dask 0.7.1
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for deisa_dask-0.7.1.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.