hpctainers 2.0.2

HPC Container Build System for Apptainer with OpenFOAM, MPI, and scientific computing focus

Build apptainer containers for your projects

This is a project to automate building HPC-ready containers (originally for OpenFOAM-based projects) using apptainer.

[!NOTE] Migration from Ansible: The previous Ansible-based system (build.yaml) has been replaced with a Python-based "container-as-code" system. The config.yaml format and container paths remain fully backward compatible so no changes are required to your existing configurations. v1 tag still has the ansible-based mechanism if you want it

[!NOTE] Brought to you by SDL Energy Conversion from in collaboration with .

• Idea
• Highlighted features
• Quick Instructions
• Migration from Ansible-based mechanism (v1 -> v2)

Idea

Automated workflows to:

• Build a base framework (eg: OpenFOAM) container (supporting various forks and versions) to run on HPCs
• Build project-specific containers that inherit from a target base container
• OpenMPI is a first-class citizen: mpirun -n 16 apptainer run container.sif "solver -parallel" should 'just work'.

Highlighted features

1. Automated, configuration-only workflows to produce containers that behave similarly across frameworks.
2. A JSON Database of container metadata, with full control at the hands of the container maintainer.
3. Maintaining definition files for your projects can be done in your own repos.
4. Loading your own repositories of base framework container definitions works seamlessly.
5. Container-as-code capabilities offering a python API to create containers as well as a REPL shell

Quick Instructions

sudo add-apt-repository -y ppa:apptainer/ppa
sudo apt install -y apptainer
curl -LsSf https://astral.sh/uv/install.sh | sh
uvx hpctainers --config config.yaml

[!TIP] The build system uses Python 3.10+ with uv for dependency management. The configuration file and base definitions provided serve as examples to build OpenFOAM containers.

The build command (by default) will:

• Create the following tree in the current working folder:

containers/
├── basic
│   ├── opencfd-openfoam.sif
│   └── ubuntu-24.04-openmpi-4.1.5.sif
└── projects
    └── test-master.sif

• Build a basic OpenMPI container containers/basic/ubuntu-24.04-openmpi-4.1.5.sif, or pull it from ghcr.io if possible

• Build a base (OpenCFD) OpenFOAM container containers/basic/opencfd-openfoam.sif, or pull it from ghcr.io if possible

• Build a test project container, to make sure MPI works alright in OpenFOAM containers

• Check the docs.md for details on how the configuration file is expected to be structured.

• Check the api-docs.md for details on how to use hpctainer's python API, and the associated REPL shell.

Here is a simplified sequence diagram describing the expected workflow:

sequenceDiagram
  actor User
  participant Build System
  participant Build Cache
  participant GHCR
  participant Docker Hub
  participant Host Env
  participant Temp File
  participant Local Build
  participant MPI Container
  participant Framework Container
  participant Project Container

  User->>Build System: Start build.py with config.yaml
  Build System->>Build Cache: Check if MPI Container cached
  Build Cache-->>Build System: Not cached or content changed
  Build System->>GHCR: Try to pull MPI Container
  GHCR-->>Build System: MPI Container not found
  Build System->>Docker Hub: Pull Ubuntu image
  Docker Hub-->>Build System: Ubuntu image pulled
  Build System->>Local Build: Build MPI Container on top of Ubuntu image
  Local Build-->>MPI Container: MPI Container created
  Build System->>Build Cache: Update cache entry
  Build System->>GHCR: Try to pull Framework Container
  GHCR-->>Build System: Framework Container not found
  Build System->>Local Build: Build Framework Container on top of MPI Container
  Local Build-->>Framework Container: Framework Container created
  Build System->>Build Cache: Update cache entry
  Note over Build System,Host Env: Environment Secrets Injection (Optional)
  Build System->>Host Env: Read environment secrets (if specified)
  Host Env-->>Build System: Return secret values
  Build System->>Temp File: Create temp file with secrets
  Temp File-->>Build System: Container-specific path
  Build System->>Local Build: Build Project Container with bind-mounted secrets
  Local Build->>Project Container: Bind-mount temp file to /tmp/hpctainers_build_env_<name>.sh
  Project Container->>Project Container: Source secrets in %post, then remove
  Local Build-->>Project Container: Project Container created (secrets removed)
  Build System->>Temp File: Clean up temp file from host
  Build System->>Build Cache: Update cache entry
  Project Container-->>User: Container ready for use (no secrets persisted)

Migration from Ansible-based mechanism (v1 -> v2)

The config.yaml format is fully backward compatible. Simply replace the build command:

# Old (Ansible)
ansible-playbook build.yaml --extra-vars="original_dir=$PWD" --extra-vars="@config.yaml"

# New (Python)
uvx hpctainers --config config.yaml