matensemble 0.2.1

An adaptive and highly asynchronous ensemble simulation workflow manager MatEnsemble (https://github.com/Q-CAD/MatEnsemble) built jointly on top of the hierarchical graph based scheduler FLUX and concurrent-futures infrastructure of python

MatEnsemble

MatEnsemble is a Python library for high-throughput workflows on HPC systems. You define a directed acyclic graph (DAG) of tasks—Python callables or executable commands—and MatEnsemble submits work through Flux, tracks completions, adapts scheduling to free CPUs and GPUs, and writes structured logs and per-chore output directories.

An optional in-tree dynopro stack supports streaming dynamics and on-the-fly analysis for advanced materials simulation workflows.

Features

• DAG-based workflows with dependencies via deferred return values (OutputReference)
• Adaptive scheduling that back-fills the allocation as tasks finish (with a non-adaptive mode when you need it)
• Two chore types: Python chores (remotely unpickled and executed by matensemble.runtime_worker) and argv-style executable chores
• Resource requests: tasks, cores per task, GPUs per task, optional MPI (pmi2) via Flux
• Observability: status.json, matensemble_workflow.log, per-chore stdout / stderr, pickle and JSON result artifacts; optional web dashboard (FastAPI on port 8000)

Documentation

Documentation (overview, architecture, tutorials, API reference):

matensemble.readthedocs.io

Installation

Containers (recommended on many clusters)

OCI images are published to GitHub Container Registry, for example:

ghcr.io/freddude2004/matensemble:baseline-vX.Y.Z

See the container packages and the Quick start in the docs for Apptainer/Singularity and site-specific notes.

Development install

From a clone of this repository:

uv sync
uv sync --group dev   # optional: docs and pytest tooling
uv run pytest

Or with pip:

pip install -e ".[flux]"

Site-specific Conda-style environment files live under scripts/ (for example scripts/baseline/environment.yaml, scripts/frontier/, scripts/perlmuter/). Align Python with 3.12+ and Flux with your center’s modules.

Quick example

from matensemble.pipeline import Pipeline

pipe = Pipeline()
pipe.exec(command=["/bin/echo", "hello from MatEnsemble"])
pipe.submit()

For Python chores, dependency graphs, and the required split between an importable chore module and a runner script, see the Tutorials.

Examples in the repository

Illustrative workflows live under example_workflows/.

Requirements and runtime

• A Flux allocation (or equivalent) on the machine where you call Pipeline.submit()
• For MPI Python or executable chores: a coherent MPI/Flux setup (e.g. PMI2) as expected by your site
• Optional: SSH port forwarding if you enable the dashboard on a compute node (see the architecture guide in the docs)

Related links

• Flux documentation
• Flux Python guide
• Slurm documentation (common front-end to batch allocations)
• LAMMPS manual (often used alongside ensemble MD workflows)

Authors

Soumendu Bagchi, Kaleb Duchesneau (see pyproject.toml for contact details).

License

BSD 3-Clause. See LICENSE.