slurmpast

How your finished Slurm jobs actually ran — and what the next one should ask for.

_{Recorded against slurmpast --demo, which is why the header reads synthetic demo data throughout. Regenerate with pip install -e ".[assets]" && python tools/demo_gif.py.}

pip install slurmpast

slurmpast              # dashboard, last 7 days
slurmpast --sizing     # what to request next time, per workload
slurmpast 51170455     # one job, every field Slurm recorded
slurmpast --demo       # no Slurm to hand? synthetic cluster
slurmpast -u alice     # someone else's history (`-u alice,bob` for several)
slurmpast --all-users  # every account on the cluster
sp                     # short alias

enter open · q back · digits jump to a row · / search · f filter · s sort · n nodes · p patterns · y copy · ? help

Hold an arrow key for three seconds and the cursor starts covering ground — top to bottom of a 28,741-row history in about twenty seconds. Let go, or press the other arrow, and it is back to one row.

Before the run, slurmate builds the request. During it, slurmwatch watches. After it, slurmpast tells you what happened and what to change.

What to request next time

Over-requesting reserves capacity nobody else can use; under-requesting kills the run. Your own history settles both.

argonne35-pretrain   test · 101 runs
  --time            raise to 10:00:00   (from 08:05:00)
      longest of 90 completed runs took 07:58:23.
  --mem             already about right
  --cpus-per-task   already about right
  #SBATCH --time=10:00:00

That workload was running on a seven-minute margin. With fewer than three usable runs it says not enough evidence instead of guessing.

The figure it measures you against — 08:05:00 above — is the limit your last run asked for, not the largest in the window. Those differ the moment you tune a request, and the window is deliberately built to span that tuning: grouping ignores resource magnitudes, so that raising --mem does not fork the history you are trying to learn from.

One job

Time, CPU and memory against the limits you asked for, then every field Slurm recorded, the log excerpts if the files are still on disk, and the findings.

Failure, across runs

seff describes one job and slurmwatch one live run. Neither can say "you submitted this 115 times and it died 99 times." Runs group into workloads, so a repeated failure — and the node it keeps landing on — is visible at a glance, with a ready-to-paste --exclude.

That --exclude is offered only when a node is worse than the rest after accounting for every other node the table tested. Without that correction a large enough history always produces a culprit: given twenty nodes with one identical true failure rate, testing each interval on its own named an innocent node more than half the time.

Correct on any cluster

sacct misreports in ways that are easy to miss: the requested memory is often blank, limits are recorded per allocation but enforced per node, and a job left in RUNNING inflates every total. slurmpast handles those, adapts to how your Slurm version spells things, and prints n/a rather than 0 for anything it cannot read.

Every trap, and the 85 fields it extracts →

As a library

The analysis modules import no third-party package.

from slurmpast import History, Sacct
from slurmpast.sizing import recommend

history = History(Sacct().history(user="you", since="now-30days"))
for advice in recommend(history.groups[0].jobs):
    print(advice.flag, advice.verdict, advice.suggestion)

--json emits everything — 103 values per job, plus 40 for every step.

Slurm 20.11.8 · Python 3.10–3.13 · Textual 0.86–8.2 · MIT