scitex-hpc

Generic SLURM dispatch — `srun` / `sbatch` / reservations / sync / poll / fetch — for any HPC cluster.

Full Documentation · uv pip install scitex-hpc[all]

Problem and Solution

Installation

pip install scitex-hpc

Demo

# Submit + watch a SLURM job (via the local scheduler config)
scitex-hpc submit-sbatch examples/00_minimal.sbatch
scitex-hpc poll-job <jobid>
scitex-hpc fetch-result <jobid> --out ./results/

# Or interactive: book a chunk, attach, work
scitex-hpc dispatch-srun --time 1:00:00 --gpus 1

graph LR
    Local["Local laptop\n(scitex-hpc CLI / Python API)"] -->|submit-sbatch| Login["HPC login node"]
    Local -->|dispatch-srun| Login
    Login -->|salloc/sbatch| Compute["Compute / GPU nodes"]
    Compute -->|stdout/err + artefacts| Sync["sync-project rsync"]
    Sync --> Local

Architecture

scitex-hpc is a thin client over the SLURM CLI (squeue, sbatch, srun, salloc). Site-specific config lives in ~/.scitex/hpc/config.yaml; nothing else is host-aware. The package is fleet-agnostic: it does not know about scitex-orochi or any specific HPC site beyond its config.

scitex_hpc/
├── _cli/                       ← `scitex-hpc` Click commands
│   ├── submit_sbatch.py        ← submit-sbatch verb
│   ├── dispatch_srun.py        ← dispatch-srun verb
│   ├── poll_job.py             ← poll-job verb
│   ├── fetch_result.py         ← fetch-result verb
│   └── sync_project.py         ← sync-project verb
├── _slurm/                     ← thin wrappers over squeue / sbatch / srun / salloc
├── _state/                     ← job-state cache (`~/.scitex/hpc/runtime/`)
└── config/                     ← YAML schema for site config

~/.scitex/hpc/                  ← user state (gitignored, host-specific)
├── config.yaml                 ← partitions, walltimes, module loads, GPU types
└── runtime/                    ← per-job caches, result-rsync metadata

1 Interfaces

from scitex_hpc import JobConfig, srun, sbatch, sync, poll_job, fetch_result

cfg = JobConfig(
    project="scitex-dsp",
    command="pip install -e '.[dev]' -q && python -m pytest tests/ -n 16",
    host="spartan", partition="sapphire",
    cpus=16, time="00:30:00", mem="64G",
)

sync(cfg)                           # 1. push local sources to the cluster
exit_code = srun(cfg)               # 2a. blocking interactive run
job_id = sbatch(cfg)                # 2b. async batch submission
print(poll_job(cfg, job_id))        # {'state': 'COMPLETED', 'exit_code': '0:0', ...}
fetch_result(cfg, job_id)           # downloads the .out file

from scitex_hpc import JobConfig, Reservation

res = Reservation.book(
    JobConfig(project="dev-pool", host="spartan", partition="cascade",
              cpus=8, mem="32G", time="7-0"),
    persistent=True,                # walltime auto-resubmit via SIGUSR1
)

res.exec("hostname")
res.exec(["python", "-m", "unittest", "discover"])
res.attach(cmd="bash")              # interactive shell on the compute node

# Look up later by friendly name (state in ~/.scitex/hpc/leases/)
res = Reservation.get("dev-pool")
res.release()                       # scancel + clear state

scitex-hpc reservations book dev-pool --host spartan --cpus 8 --mem 32G --time 7-0 --persistent
scitex-hpc reservations list
scitex-hpc reservations exec dev-pool 'hostname'
scitex-hpc reservations attach dev-pool
scitex-hpc reservations cancel dev-pool

Defaults & overrides

Every JobConfig field has a SCITEX_HPC_* env-var override:

Resolution priority: explicit JobConfig field → env var → built-in default.

Walltime auto-resubmit (persistent=True)

When persistent=True, scitex-hpc:

1. Adds #SBATCH --signal=B:USR1@3600 so SLURM signals the script 1 h before walltime.
2. Wraps the sbatch body with a SIGUSR1 trap that calls sbatch "$0" to resubmit itself.
3. The friendly name (dev-pool) stays stable across resubmits; the SLURM job_id changes.

res = Reservation.get("dev-pool")
res.refresh()                       # squeue --user --name=dev-pool
res.exec("...")                     # uses the new job_id

SLURM's documented signaling mechanism — no custom daemon. Compatible with HPC policies that ban persistent user-space daemons. SSH ControlMaster pooling on the calling host amortizes per-exec() handshake cost.

Part of SciTeX

scitex-hpc is part of SciTeX. Install via the umbrella with pip install scitex[hpc] to use as scitex.hpc (Python) or scitex hpc ... (CLI).

Four Freedoms for Research

0. The freedom to run your research anywhere — your machine, your terms.
1. The freedom to study how every step works — from raw data to final manuscript.
2. The freedom to redistribute your workflows, not just your papers.
3. The freedom to modify any module and share improvements with the community.

AGPL-3.0 — because we believe research infrastructure deserves the same freedoms as the software it runs on.