dagster-slurm 1.14.0

dagster integration with slurm

dagster-slurm

Integrating dagster to orchestrate slurm jobs for HPC systems and frameworks for scaling compute like ray for a better developer experience on supercomputers.

dagster-slurm lets you take the same Dagster assets from a laptop to a Slurm-backed supercomputer with minimal configuration changes.

An European sovereign GPU cloud does not come out of nowhere maybe this project can support making HPC systems more accessible.

Basic example

https://github.com/ascii-supply-networks/dagster-slurm/tree/main/examples

prerequisites

• installation of pixi: https://pixi.sh/latest/installation/ curl -fsSL https://pixi.sh/install.sh | sh
• pixi global install git
• a container runtime like docker or podman; for now we assume docker compose is available to you. You could absolutely also use nerdctl or something similar.

usage

Example

git clone https://github.com/ascii-supply-networks/dagster-slurm.git
docker compose up
cd dagster-slurm/examples

local execution

Execute without slurm.

• Small data
• Rapid local prototyping

pixi run start

go to http://localhost:3000 and you should see the dagster webserver running.

docker local execution

• Test everything works on SLURM
• Still small data
• Mainly used for developing this integration

Ensure you have a .env file with the following content:

SLURM_EDGE_NODE_HOST=localhost
SLURM_EDGE_NODE_PORT=2223
SLURM_EDGE_NODE_USER=submitter
SLURM_EDGE_NODE_PASSWORD=submitter
SLURM_DEPLOYMENT_BASE_PATH=/home/submitter/pipelines/deployments

pixi run start-staging

go to http://localhost:3000 and you should see the dagster webserver running.

prod docker local execution

• Test everything works on SLURM
• Still small data
• Mainly used for developing this integration
• This target instead supports a faster startup of the job

Ensure you have a .env file with the following content:

SLURM_EDGE_NODE_HOST=localhost
SLURM_EDGE_NODE_PORT=2223
SLURM_EDGE_NODE_USER=submitter
SLURM_EDGE_NODE_PASSWORD=submitter
SLURM_DEPLOYMENT_BASE_PATH=/home/submitter/pipelines/deployments

# see the JQ command below for dynamically setting this
# CI_DEPLOYED_ENVIRONMENT_PATH=/home/submitter/pipelines/deployments/<<<your deployment >>>

# we assume your CI-CD pipelines would out of band perform the deployment of the environment
# this allows your jobs to start up faster
pixi run deploy-prod-docker

cat deplyyment_metadata.json
export CI_DEPLOYED_ENVIRONMENT_PATH="$(jq -er '.deployment_path' foo.json)"

pixi run start-prod-docker

go to http://localhost:3000 and you should see the dagster webserver running.

real HPC supercomputer execution

• Targets clusters like VSC-5 (Austrian Scientific Computing (ASC)) and Leonardo (CINECA).
• Assets run against the real scheduler, so ensure the account has queue access and quotas.

Create a .env file with the edge-node credentials and select the site profile:

# example for VSC-5
SLURM_EDGE_NODE_HOST=vsc5.vsc.ac.at
SLURM_EDGE_NODE_PORT=22
SLURM_EDGE_NODE_USER=<<your_user>>
SLURM_EDGE_NODE_PASSWORD=<<your_password>>
SLURM_EDGE_NODE_JUMP_HOST=vmos.vsc.ac.at
SLURM_EDGE_NODE_JUMP_USER=<<your_user>>
SLURM_EDGE_NODE_JUMP_PASSWORD=<<your_password>>
SLURM_DEPLOYMENT_BASE_PATH=/home/<<your_user>>/pipelines/deployments
SLURM_PARTITION=zen3_0512
SLURM_QOS=zen3_0512_devel
SLURM_RESERVATION=dagster-slurm_21
SLURM_SUPERCOMPUTER_SITE=vsc5
DAGSTER_DEPLOYMENT=staging_supercomputer

If your account relies on passwords (or passwords + OTP), provide them for both the jump host and the final login node. The automation will answer the standard prompts; any time-based OTP still has to be supplied interactively once per validity window. When an extra prompt appears, Dagster writes Enter ... for <host>: to your terminal (via /dev/tty). Enter the code there to continue.

TTY allocation is handled automatically for password-based sessions, so you do not need to set SLURM_EDGE_NODE_FORCE_TTY unless your centre requires it explicitly.

With the variables in place, validate connectivity and job submission using the staging supercomputer profile:

pixi run start-staging-supercomputer

Staging mode packages dependencies on demand. Expect the first asset run to upload a new environment bundle before dispatching the Slurm job.

For production you should pre-build and upload the execution environment via your CI/CD pipeline (see examples/scripts/deploy_environment.py). Capture the output path and expose it to Dagster as CI_DEPLOYED_ENVIRONMENT_PATH:

python scripts/deploy_environment.py --platform linux-64  # run from CI
# -> produces deployment_metadata.json with "deployment_path"

export CI_DEPLOYED_ENVIRONMENT_PATH=/home/submitter/pipelines/deployments/prod-env-20251018
export DAGSTER_DEPLOYMENT=production_supercomputer
pixi run start-production-supercomputer

If CI_DEPLOYED_ENVIRONMENT_PATH is missing, the production profile will refuse to start to prevent accidental live builds on the cluster.

To confirm a submission landed on the expected queue, run:

ssh -J <<your_user>>@vmos.vsc.ac.at <<your_user>>@vsc5.vsc.ac.at \
  "squeue -j <jobid> -o '%i %P %q %R %T'"

The Partition, QOS, and Reservation columns should match your .env.

Ray launcher overrides

If your cluster needs OS-level tweaks before Ray starts (for example, higher file-descriptor limits), configure pre_start_commands on the Ray launcher. To pass extra arguments to ray start (for example, disabling the dashboard), use ray_start_args:

"launchers": {
    "ray": {
        "pre_start_commands": [
            "ulimit -n 65536",
        ],
    },
}

You can attach these overrides inside a site profile (see SUPERCOMPUTER_SITE_OVERRIDES in the example resources).

contributing

See the contributing guide for how to contribute! Help building and maintaining this project is welcome.