slurm-job-analyzer 0.1.1

Parse Slurm sacct exports: GPU waste reports, utilization summaries, and queue analytics for HPC clusters

slurm-job-analyzer

Parse Slurm sacct exports to identify GPU waste, measure utilization efficiency, and produce per-partition queue analytics — without touching the scheduler.

Works with any Slurm cluster: NVIDIA (H100/H200/A100), AMD (MI300X), on-prem or cloud.

---

Why this exists

HPC clusters routinely see 30–60 % of allocated GPU-hours go unused — jobs request 8 GPUs but run at 5 % utilization. Standard sacct output is hard to parse at scale. slurm-analyze turns raw sacct TSV exports into actionable reports in one command.

---

Install

pip install slurm-job-analyzer

Requires Python 3.10+. No CUDA, no cluster access needed — works on any machine with a sacct export file.

---

Quick start

1. Export from Slurm

sacct \
  --starttime=2024-01-01 \
  --endtime=2024-12-31 \
  --format=JobID,JobName,Partition,State,Elapsed,AllocCPUS,AllocTRES,ReqMem,MaxRSS,NodeList \
  --parsable2 \
  > jobs.tsv

2. Partition utilization summary

slurm-analyze sacct --input jobs.tsv --out results/summary.csv

Partition summary (1423 jobs parsed):
  partition   jobs  total_wall_h  avg_wall_min  completed_pct  total_gpu_hours  gpu_nodes
  ---------   ----  ------------  ------------  -------------  ---------------  ---------
  gpu          847        4821.3          34.2         0.8900          9821.44         12
  gpu-debug    312         183.1           3.6         0.9400           124.80          4
  preempt      264        1204.7          27.4         0.7100          2340.12          8

3. GPU waste report

Combine sacct with nvidia-smi snapshots for per-job efficiency verdicts:

slurm-analyze gpu-waste \
  --input jobs.tsv \
  --gpu-log snapshots.csv \
  --idle-threshold 20 \
  --out results/gpu_waste.csv

GPU waste report (847 GPU jobs):
  job_id  job_name      alloc_gpus  elapsed_h  gpu_hours  avg_util_pct  verdict
  ------  --------      ----------  ---------  ---------  ------------  -------
  10042   train_llama   8           12.3        98.4       87.2          EFFICIENT
  10051   eval_run      4            2.1         8.4        6.1          IDLE_WASTE
  10063   finetune      8            5.0        40.0       52.3          UNDERUTILIZED

Summary:
  Total GPU-hours allocated:   9821.44 h
  EFFICIENT (>=80% util):      312
  UNDERUTILIZED (20-79%):      401
  IDLE_WASTE (<20% util):       98
  MEM_OVERALLOC (mem <50%):     36

4. Queue / partition resource summary

slurm-analyze queue-wait --input jobs.tsv

5. GPU utilization timeline

slurm-analyze timeline --job 10042 --snapshots snapshots.csv

---

GPU snapshot format

Collect snapshots per node with:

nvidia-smi \
  --query-gpu=timestamp,name,utilization.gpu,temperature.gpu,power.draw,memory.used,memory.total \
  --format=csv,noheader,nounits \
  --loop=10

Then transform into the expected CSV with columns: timestamp_s, node, gpu_index, utilization_pct, temperature_c, power_w, memory_used_mib, memory_total_mib

---

Python API

from slurm_analyze import read_sacct_tsv, gpu_waste_report, summarize_by_partition
from pathlib import Path

rows = read_sacct_tsv(Path("jobs.tsv"))
summary = summarize_by_partition(rows)

for partition in summary:
    print(f"{partition['partition']}: {partition['total_gpu_hours']:.1f} GPU-hours")

reports = gpu_waste_report(rows, snapshots=[], idle_threshold_pct=20.0)
wasted = [r for r in reports if r.verdict == "IDLE_WASTE"]
print(f"{len(wasted)} jobs wasted GPU time")

---

CLI reference

slurm-analyze --version

slurm-analyze sacct       --input FILE [--out FILE]
slurm-analyze gpu-waste   --input FILE [--gpu-log FILE] [--out FILE] [--idle-threshold N]
slurm-analyze queue-wait  --input FILE [--out FILE]
slurm-analyze timeline    --job JOB_ID --snapshots FILE

---

Development

git clone https://github.com/namanadep/slurm-job-analyzer
cd slurm-job-analyzer
pip install -e ".[dev]"
pytest

---

License

MIT — see LICENSE.