pytest-slurm

A pytest plugin providing a Slurm cluster testing environment using vHPC.

About eXact lab

This project is open-sourced by eXact lab S.r.l., a consultancy specializing in scientific and high-performance computing solutions. We help organizations optimize their computational workflows, implement scalable HPC infrastructure, and accelerate scientific research through tailored technology solutions.

Need HPC expertise? Contact us for consulting services in scientific computing, cluster optimization, and performance engineering.

Security

Running the test suite requires Docker access and pulls container images from ghcr.io/exactlab (the vHPC headnode and worker images). These images use hardcoded credentials intended for local testing only. Do not point the slurm_ssh fixtures at production infrastructure, and do not reuse these credentials outside a local test environment.

Quick Start

Installation

pip install pytest-slurm

Usage

The plugin provides two fixtures for interacting with a Slurm cluster:

def test_slurm_commands(slurm):
    """Test using Docker exec (faster)."""
    result = slurm.run("sinfo")
    assert result.exit_code == 0

def test_slurm_ssh(slurm_ssh):
    """Test using SSH connection (realistic)."""
    result = slurm_ssh.run("srun echo 'Hello Slurm'")
    assert result.exit_code == 0

def test_stdin_support(slurm):
    """Test commands with stdin input."""
    result = slurm.run("cat", input="hello world")
    assert result.stdout.decode().strip() == "hello world"

def test_popen_interface(slurm):
    """Test subprocess-like popen interface."""
    popen = slurm.popen("cat")
    stdout, stderr = popen.communicate(b"hello world")
    assert popen.returncode == 0
    assert stdout.strip() == b"hello world"

Contents

• Usage
• Fixtures
• Testing code that runs on the cluster
• Configuration changes and rebuilding
• Project isolation
• Keep alive mode / development mode
• License

Fixtures

• slurm_raw: Raw connection parameters (container name, host, port, credentials)
• slurm: Local connection via Docker exec
• slurm_ssh: SSH connection to headnode
• slurm_ssh_privkey: SSH private key from headnode
• slurm_python: Path to Python interpreter in the cluster's virtual environment
• slurm_project: Slurm connection with automatic project mounting and editable install

The slurm_raw fixture provides the underlying connection details as a dictionary with keys: container_name, host, port, username, password, and partiton.

The slurm_python fixture returns /opt/venv/bin/python, which is the path to the Python interpreter within the shared virtual environment accessible from both the headnode and workers. Use this when submitting jobs that execute Python code:

def test_python_job(slurm, slurm_python):
    """Submit a Python job to the cluster."""
    result = slurm.run(f"srun {slurm_python} -c 'print(\"Hello from Python\")'")
    assert result.exit_code == 0
    assert b"Hello from Python" in result.stdout

Both slurm and slurm_ssh fixtures provide:

• .run(command, input=None) method returning a CommandResult with exit_code, stdout, and stderr attributes
• .popen(command) method returning a subprocess-like object with .communicate(input=None) method

Testing code that runs on the cluster

When testing Python code that needs to execute within Slurm jobs, use the slurm_project fixture to automatically mount your project into the cluster and install it as editable:

def test_my_package_on_cluster(slurm_project, slurm_python):
    """Test package code running in a Slurm job."""
    result = slurm_project.run(
        f"srun {slurm_python} -c 'import mypackage; mypackage.run()'"
    )
    assert result.exit_code == 0

The slurm_project fixture automatically:

• Mounts your project directory at /opt/project (read-write on headnode, read-only on workers)
• Installs your package in editable mode via pip install -e /opt/project
• Returns the same interface as the slurm fixture

Auto-detection: When any test uses slurm_project, mounting is enabled automatically. You can also manually enable it with --slurm-mount-project:

pytest --slurm-mount-project

Configuration changes and rebuilding

The plugin automatically tracks configuration changes and rebuilds containers when needed. Configuration includes the compose file structure and packages.yml content. When a change is detected, containers are automatically rebuilt with the new configuration.

To force a rebuild regardless of changes:

pytest --slurm-rebuild

This is useful when containers are in an inconsistent state or when you want to ensure a clean environment.

Project isolation

Each project automatically gets its own isolated cluster based on the project directory name. For example, a project in /home/user/myproject will use containers with the name pytest_slurm_myproject. This allows multiple projects to have their own clusters without conflicts.

The project name is automatically sanitised (lowercase, alphanumeric plus underscores/hyphens) to ensure Docker Compose compatibility.

SSH ports are dynamically allocated starting from 2222 to prevent conflicts when multiple projects run simultaneously. Each cluster gets its own unique port.

Keep alive mode / development mode

For faster development iteration, you can start the Slurm cluster once and reuse it across multiple test runs:

# Start cluster and keep it alive
pytest --slurm-keep-alive

# In another terminal, run tests (reuses existing containers)
pytest tests/

# Run tests multiple times without container startup overhead
pytest tests/test_specific.py

Typical Development Workflow:

1. Start keep-alive mode: pytest --slurm-keep-alive
2. Run your tests repeatedly in another terminal
3. Stop keep-alive mode with Ctrl+C when done

This reduces test execution time during development by eliminating the container startup overhead at the cost of full test isolation, as the virtual cluster is persisted across test sessions.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Copyright (c) 2025 Jacopo Nespolo, eXact lab S.r.l.