ADIOS4DOLFINx - A framework for checkpointing in DOLFINx

ADIOS4DOLFINx is an extension for DOLFINx to checkpoint meshes, meshtags and functions using ADIOS 2.

The code uses the ADIOS2 Python-wrappers to write DOLFINx objects to file, supporting N-to-M (recoverable) and N-to-N (snapshot) checkpointing. See: Checkpointing in DOLFINx - FEniCS 23 or the examples in the Documentation for more information.

For scalability, the code uses MPI Neighbourhood collectives for communication across processes.

Statement of Need

As the usage of high performance computing clusters increases, more and more large-scale, long-running simulations are deployed. The need for storing intermediate solutions from such simulations are crucial, as the HPC system might crash, or the simulation might crash or exceed the alloted computational budget. Having a checkpoint of related variables, such as the solutions to partial differential equations (PDEs) is therefore essential. The adios4dolfinx library extends the DOLFINx computational framework for solving PDEs with checkpointing functionality, such that immediate solutions and mesh information can be stored and re-used in another simulation.

Installation

Compatibility with DOLFINx:

• ADIOS4DOLFINx v0.10.0 is compatible with DOLFINx v0.10.x
• ADIOS4DOLFINx v0.9.6 is compatible with DOLFINx v0.9.x
• ADIOS4DOLFINx v0.8.1 is compatible with DOLFINx v0.8.x
• ADIOS4DOLFINx v0.7.3 is compatible with DOLFINx v0.7.x

Dependencies

The library depends on the Python-interface of DOLFINx and an MPI-build of ADIOS2. Therefore ADIOS2 should not be install through PYPI/pip, but has to be installed through Conda, Spack or from source.

[!IMPORTANT]
ADIOS2<2.10.2 does not work properly with numpy>=2.0.0. Everyone is advised to use the newest version of ADIOS2. This is for instance available through conda or the ghcr.io/fenics/dolfinx/dolfinx:nightly Docker-image.

Spack

ADIOS4DOLFINx is a spack package which can be installed with

spack add py-adios4dolfinx ^py-fenics-dolfinx+petsc4py+slepc4py
spack concretize
spack install

once you have downloaded spack and set up a new environment, as described in Spack: Installation notes. To ensure that the spack packages are up to date, please call

spack repo update builtin

prior to concretizing.

Docker

An MPI build of ADIOS2 is installed in the official DOLFINx containers, and thus there are no additional dependencies required to install adios4dolfinx on top of DOLFINx in these images.

Create a Docker container, named for instance dolfinx-checkpoint. Use the nightly tag to get the main branch of DOLFINx, or stable to get the latest stable release

docker run -ti -v $(pwd):/root/shared -w /root/shared --name=dolfinx-checkpoint ghcr.io/fenics/dolfinx/dolfinx:nightly

For the latest version compatible with nightly (with the ability to run the test suite), use

python3 -m pip install adios4dolfinx[test]@git+https://github.com/jorgensd/adios4dolfinx@main

If you are using the stable image, you can install adios4dolfinx from PYPI with

python3 -m pip install adios4dolfinx[test]

This docker container can be opened with

docker container start -i dolfinx-checkpoint

at a later instance

Conda

[!NOTE]
Conda supports the stable release of DOLFINx, and thus the appropriate version should be installed, see the section above for more details.

Following is a minimal recipe of how to install adios4dolfinx, given that you have conda installed on your system.

conda create -n dolfinx-checkpoint python=3.10
conda activate dolfinx-checkpoint
conda install -c conda-forge adios4dolfinx

[!NOTE] Remember to download the appropriate version of adios4dolfinx from Github adios4dolfinx: Releases

To run the test suite, you should also install ipyparallel, pytest and coverage, which can all be installed with conda

conda install -c conda-forge ipyparallel pytest coverage

Functionality

DOLFINx

• Reading and writing meshes, using adios4dolfinx.read/write_mesh
• Reading and writing meshtags associated to meshes adios4dolfinx.read/write_meshtags
• Reading checkpoints for any element (serial and parallel, arbitrary number of functions and timesteps per file). Use adios4dolfinx.read/write_function.
• Writing standalone function checkpoints relating to "original meshes", i.e. meshes read from XDMFFile. Use adios4dolfinx.write_function_on_input_mesh for this.
• Store mesh partitioning and re-read the mesh with this information, avoiding calling SCOTCH, Kahip or Parmetis.

[!IMPORTANT]
For checkpoints written with write_function to be valid, you first have to store the mesh with write_mesh to the checkpoint file.

[!IMPORTANT]
A checkpoint file supports multiple functions and multiple time steps, as long as the functions are associated with the same mesh

[!IMPORTANT]
Only one mesh per file is allowed

Example Usage

The repository contains many documented examples of usage, in the docs-folder, including

• Reading and writing mesh checkpoints
• Storing mesh partitioning data
• Writing mesh-tags to a checkpoint
• Reading and writing function checkpoints
• Checkpoint on input mesh Further examples can be found at ADIOS4DOLFINx examples

Backwards compatibility

[!WARNING] If you are using v0.7.2, you are advised to upgrade to v0.7.3, as it contains som crucial fixes for openmpi.

Legacy DOLFIN

Only checkpoints for Lagrange or DG functions are supported from legacy DOLFIN

• Reading meshes from the DOLFIN HDF5File-format
• Reading checkpoints from the DOLFIN HDF5File-format (one checkpoint per file only)
• Reading checkpoints from the DOLFIN XDMFFile-format (one checkpoint per file only, and only uses the .h5 file)

See the API for more information.

Testing

This library uses pytest for testing. To execute the tests, one should first install the library and its dependencies, as listed above. Then, can execute all tests by calling

python3 -m pytest .

Testing against data from legacy dolfin

Some tests check the capability of reading data created with the legacy version of DOLFIN. To create this dataset, start a docker container with legacy DOLFIN, for instance:

docker run -ti -v $(pwd):/root/shared -w /root/s
hared --rm ghcr.io/scientificcomputing/fenics:2024-02-19

Then, inside this container, call

python3 ./tests/create_legacy_data.py --output-dir=legacy

Testing against data from older versions of ADIOS4DOLFINx

Some tests check the capability to read data generated by adios4dolfinx<0.7.2. To generate data for these tests use the following commands:

docker run -ti -v $(pwd):/root/shared -w /root/shared --rm ghcr.io/fenics/dolfinx/dolfinx:v0.7.3

Then, inside the container, call

python3 -m pip install adios4dolfinx==0.7.1
python3 ./tests/create_legacy_checkpoint.py --output-dir=legacy_checkpoint

Long term plan

The long term plan is to get this library merged into DOLFINx (rewritten in C++ with appropriate Python-bindings).