slurmforge 1.2.3

Experiment orchestration toolkit for Slurm-based training and evaluation workflows.

slurmforge

slurmforge is a Slurm-native stage-batch system for AI training and evaluation workflows.

It focuses on a small CLI surface:

sforge init
sforge init --list-templates
sforge init --template train-eval --output ./demo --force
cd demo
sforge validate --config experiment.yaml
sforge estimate --config experiment.yaml
sforge plan train --config experiment.yaml --dry-run=full --output plan.audit.json
sforge plan eval --config experiment.yaml --checkpoint /path/to/model.pt --input-name model_input
sforge plan run --config experiment.yaml
sforge train --config experiment.yaml --dry-run=full
sforge eval --config experiment.yaml --checkpoint /path/to/model.pt
sforge run --config experiment.yaml
sforge pipeline resume /path/to/pipeline-root
sforge status --from /path/to/root --reconcile
sforge status --from /path/to/root --reconcile --sacct-profile portable
sforge resubmit --from /path/to/root --stage eval --query state=failed

sforge run pipelines auto-advance to completion via a dependency-free watchdog (configurable under orchestration.control.auto_advance). sforge pipeline resume advances a pipeline once on demand — useful when auto-advance is disabled or driven externally by cron/scrontab.

Install

python -m venv .venv
source .venv/bin/activate
python -m pip install -e '.[dev]'

Start

Create a starter project instead of writing YAML from scratch:

sforge init

For scripts or CI, choose a template explicitly:

sforge init --template train-eval --output ./demo --force

This writes ./demo/experiment.yaml, ./demo/CONFIG.sforge.md, ./demo/README.sforge.md, and the template's stage scripts.

Available starter templates:

• train-eval: train produces a checkpoint; eval consumes the upstream output.
• train-only: one train stage with a checkpoint output.
• eval-checkpoint: one eval stage that consumes an explicit checkpoint path.

The generated train.py and eval.py are structured as integration scaffolds:

• SECTION A - SlurmForge contract: injected CLI args and environment contract.
• SECTION B - Your model code: model construction, data loading, training, and eval logic to replace.
• SECTION C - Output contract: checkpoint and metrics files declared by the YAML.

Minimal Workflow

sforge validate --config experiment.yaml
sforge run --config experiment.yaml --dry-run=full
sforge run --config experiment.yaml --emit-only
sforge run --config experiment.yaml
sforge status --from ./runs/<project>/<experiment>/<pipeline-root> --reconcile

status --reconcile uses sacct profile fallback by default and can be pinned with --sacct-profile or --sacct-fields for clusters with different sacct field support.

Use sforge train for train-only configs and sforge eval --checkpoint /path/to/model.pt for eval-only configs.

Docs

• Quickstart
• Config Reference
• Operations
• Internals
• Record Contract

Development

ruff check src tests
pytest -q