scitex-hpc 0.8.1

Generic SLURM dispatch (srun, sbatch, sync, poll, fetch) for the SciTeX ecosystem — login nodes never run compute

scitex-hpc

Generic SLURM dispatch — `srun` / `sbatch` / reservations / sync / poll / fetch — for any HPC cluster.

Full Documentation · uv pip install scitex-hpc[all]

---

Problem and Solution

#	Problem	Solution
1	Login nodes silently run compute — sysadmins kill stray processes, jobs die unannounced	Every command wrapped in srun / sbatch via login-shell SSH so SLURM modules load correctly
2	Queue wait dominates iteration for short multi-agent / dev workflows	Reservation.book(..., persistent=True) — book a node once, exec() many short commands inside the same allocation; SIGUSR1 trap auto-resubmits at walltime
3	Per-cluster knob soup (partition, cpus, time, mem) repeated in every job script	SCITEX_HPC_* env overrides + JobConfig defaults for spartan/sapphire — script once, deploy anywhere

Installation

pip install scitex-hpc

Demo

# Submit + watch a SLURM job (via the local scheduler config)
scitex-hpc submit-sbatch examples/00_minimal.sbatch
scitex-hpc poll-job <jobid>
scitex-hpc fetch-result <jobid> --out ./results/

# Or interactive: book a chunk, attach, work
scitex-hpc dispatch-srun --time 1:00:00 --gpus 1

graph LR
    Local["Local laptop\n(scitex-hpc CLI / Python API)"] -->|submit-sbatch| Login["HPC login node"]
    Local -->|dispatch-srun| Login
    Login -->|salloc/sbatch| Compute["Compute / GPU nodes"]
    Compute -->|stdout/err + artefacts| Sync["sync-project rsync"]
    Sync --> Local

Architecture

scitex-hpc is a thin client over the SLURM CLI (squeue, sbatch, srun, salloc). Site-specific config lives in ~/.scitex/hpc/config.yaml; nothing else is host-aware. The package is fleet-agnostic: it does not know about scitex-orochi or any specific HPC site beyond its config.

scitex_hpc/
├── _cli/                       ← `scitex-hpc` Click commands
│   ├── submit_sbatch.py        ← submit-sbatch verb
│   ├── dispatch_srun.py        ← dispatch-srun verb
│   ├── poll_job.py             ← poll-job verb
│   ├── fetch_result.py         ← fetch-result verb
│   └── sync_project.py         ← sync-project verb
├── _slurm/                     ← thin wrappers over squeue / sbatch / srun / salloc
├── _state/                     ← job-state cache (`~/.scitex/hpc/runtime/`)
└── config/                     ← YAML schema for site config

~/.scitex/hpc/                  ← user state (gitignored, host-specific)
├── config.yaml                 ← partitions, walltimes, module loads, GPU types
└── runtime/                    ← per-job caches, result-rsync metadata

1 Interfaces

Python API ⭐⭐⭐ (primary)

from scitex_hpc import JobConfig, srun, sbatch, sync, poll_job, fetch_result

cfg = JobConfig(
    project="scitex-dsp",
    command="pip install -e '.[dev]' -q && python -m pytest tests/ -n 16",
    host="spartan", partition="sapphire",
    cpus=16, time="00:30:00", mem="64G",
)

sync(cfg)                           # 1. push local sources to the cluster
exit_code = srun(cfg)               # 2a. blocking interactive run
job_id = sbatch(cfg)                # 2b. async batch submission
print(poll_job(cfg, job_id))        # {'state': 'COMPLETED', 'exit_code': '0:0', ...}
fetch_result(cfg, job_id)           # downloads the .out file

Reservations (book once, exec many)

from scitex_hpc import JobConfig, Reservation

res = Reservation.book(
    JobConfig(project="dev-pool", host="spartan", partition="cascade",
              cpus=8, mem="32G", time="7-0"),
    persistent=True,                # walltime auto-resubmit via SIGUSR1
)

res.exec("hostname")
res.exec(["python", "-m", "unittest", "discover"])
res.attach(cmd="bash")              # interactive shell on the compute node

# Look up later by friendly name (state in ~/.scitex/hpc/leases/)
res = Reservation.get("dev-pool")
res.release()                       # scancel + clear state

CLI ⭐⭐

scitex-hpc reservations book dev-pool --host spartan --cpus 8 --mem 32G --time 7-0 --persistent
scitex-hpc reservations list
scitex-hpc reservations exec dev-pool 'hostname'
scitex-hpc reservations attach dev-pool
scitex-hpc reservations cancel dev-pool

Defaults & overrides

Every JobConfig field has a SCITEX_HPC_* env-var override:

Field	Default	Env override
host	spartan	SCITEX_HPC_HOST
partition	sapphire	SCITEX_HPC_PARTITION
cpus	16	SCITEX_HPC_CPUS
time	00:20:00	SCITEX_HPC_TIME
mem	128G	SCITEX_HPC_MEM
remote_base	~/proj	SCITEX_HPC_REMOTE_BASE

Resolution priority: explicit JobConfig field → env var → built-in default.

Walltime auto-resubmit (persistent=True)

When persistent=True, scitex-hpc:

1. Adds #SBATCH --signal=B:USR1@3600 so SLURM signals the script 1 h before walltime.
2. Wraps the sbatch body with a SIGUSR1 trap that calls sbatch "$0" to resubmit itself.
3. The friendly name (dev-pool) stays stable across resubmits; the SLURM job_id changes.

res = Reservation.get("dev-pool")
res.refresh()                       # squeue --user --name=dev-pool
res.exec("...")                     # uses the new job_id

SLURM's documented signaling mechanism — no custom daemon. Compatible with HPC policies that ban persistent user-space daemons. SSH ControlMaster pooling on the calling host amortizes per-exec() handshake cost.

Part of SciTeX

scitex-hpc is part of SciTeX. Install via the umbrella with pip install scitex[hpc] to use as scitex.hpc (Python) or scitex hpc ... (CLI).

Four Freedoms for Research

0. The freedom to run your research anywhere — your machine, your terms.
1. The freedom to study how every step works — from raw data to final manuscript.
2. The freedom to redistribute your workflows, not just your papers.
3. The freedom to modify any module and share improvements with the community.

AGPL-3.0 — because we believe research infrastructure deserves the same freedoms as the software it runs on.

---