clust — see who is using the SLURM cluster, and how much of your quota is left

clust reads SLURM directly and prints a dashboard: how much of the cluster is busy, how each QOS stands against its GPU quota, and which users are holding what. A full refresh takes well under a second on a cluster with thousands of nodes and jobs.

All examples below are rendered from a fictional cluster.

Install

pip install slurmclust     # installs the `clust` command

No third-party Python dependencies. Needs scontrol and squeue on PATH; sacctmgr too if you want quotas. Tested against SLURM 25.05; the squeue --Format fields it relies on (tres-alloc, PendingTime, Account) have been available since SLURM 20.11.

Usage

clust                     # your QOS and accounts: quota, usage, top consumer
clust -q gpu_vision_high  # per-user breakdown of one QOS — who do I ping?
clust -a                  # whole cluster: partitions, free capacity, top users, queue
clust -p gpu,gpu-big      # cluster view restricted to some partitions
clust -A vision           # focus on an account instead of your own
clust -u someone          # resolve QOS access as if you were another user
clust -g a100             # restrict to one GPU type (case-insensitive substring)
clust -f bigmem           # restrict to nodes with given features
clust -w 10               # refresh every 10s
clust --ascii             # bars without block characters
clust --debug-timing      # how long each SLURM call took

Default view — your QOS

With no arguments, clust resolves the QOS you can actually submit to (via sacctmgr show assoc user=$USER) and shows each against the GPU quota SLURM enforces on it (GrpTRES gres/gpu):

CLUSTER  mycluster  96 nodes · 672 GPUs · 9 running · 4 queued                                12:50:10
  ███████████████████████████▓▒░······   79.6%   535 used · 8 blocked · 18 drain · 8 down · 103 free

YOUR QOS   user alice · quota is the QOS GrpTRES gres/gpu limit SLURM enforces
  QOS               PART      QUOTA   USED   %QUOTA   PEND   USERS   TOP CONSUMER
  gpu_vision_high   gpu-big     256    224      88%     96       3   alice (128)    ██████████████████··
  gpu_shared        gpu         512    128      25%     16       2   dave (80)      █████···············
  gpu_lowest        gpu         672     16       2%      -       1   dave (16)      ····················

ACCOUNTS
  ACCOUNT   QUOTA   USED   %QUOTA   PEND   USERS
  vision      320    272      85%     96       3   ███████████████████████████████·····

QOS rows are sorted by how close they are to their quota, so whatever is contended shows up first. Usage can exceed 100% where SLURM lets a QOS overflow its group limit.

Drill-down — who is holding the GPUs

$ clust -q gpu_vision_high

gpu_vision_high  ████████████████████████████████····  224 / 256 GPUs  (88%)   +96 pending
  USER    GPUS   SHARE   JOBS   NODES   OLDEST   PEND   WAITING
  alice    128     57%      1      16    2d06h     32       30m   ███████████████···········
  bob       64     29%      1       8   18h22m     64     2h35m   ███████···················
  carol     32     14%      1       4    3d01h      -         -   ████······················

OLDEST is the age of the user's longest-running job, PEND the GPUs they have queued behind it, and WAITING how long that queued work has been sitting.

Cluster view

$ clust -a

PARTITIONS
  PARTITION   NODES   GPUS   USED   FREE   BLOCKED   DRAIN   DOWN   UTIL
  gpu-big        60    480    387     67         0      18      8    81%   ██████████████████████▒░····
  gpu            24    192    148     36         8       0      0    77%   ██████████████████████▓·····
  cpu            12      –      –      –         –       –      –      –   cpu only

FREE CAPACITY   GPUs obtainable in whole chunks of N per node
  PARTITION   ≥1 GPU   ≥2 GPU   ≥4 GPU   ≥8 GPU
  gpu-big         67       60       60       32
  gpu             36       36       24        0

TOP USERS   by allocated GPUs · 6 active · bars relative to the leader
  USER    ACCOUNT   GPUS   SHARE   JOBS   NODES   OLDEST   PEND
  alice   vision     176   36.1%      2      22    2d06h     32   ██████████████████████████████
  dave    speech      96   19.7%      2      12    1d04h     16   ████████████████··············
  erin    nlp         96   19.7%      1      12   11h48m    128   ████████████████··············
  bob     vision      64   13.1%      1       8   18h22m     64   ███████████···················

QUEUE   4 pending jobs
  PARTITION   JOBS   GPUS REQ   USERS   LONGEST WAIT   TOP WAITER
  gpu-big        3        224       3          3d03h   erin (128)
  gpu            1         16       1            10m   dave (16)

Columns that are zero everywhere (BLOCKED, DRAIN, DOWN, PEND) are hidden, so a healthy cluster shows a narrower table. Bars shrink and then drop out entirely on narrow terminals — you lose the decoration, never the numbers.

How capacity is counted

Every GPU falls into exactly one bucket, so the columns always add up:

blocked is the "virtual GPU" idea this tool was built around: a job that takes many CPUs and no GPUs still makes the rest of the node unusable. A node can only offer floor(free_cpus / cpus_per_gpu) GPUs, whatever its idle GPU count says.

FREE CAPACITY reports GPUs obtainable in whole chunks per node — ≥8 GPU is what a job asking for a full node can actually get. The chunk sizes shown are derived from the largest node in the selection.

Capacity comes from node state only; per-user, per-QOS and per-account attribution comes from jobs only. QOS and account usage is reported in raw allocated GPUs, because that is what SLURM's GrpTRES limits are enforced against.

Performance

clust deliberately avoids SLURM's --json output, which serialises the entire object graph. Measured on a 3,464-node / 4,265-job cluster:

All four SLURM calls are issued in parallel, so wall clock is the slowest one. Use --debug-timing for the breakdown.

Handy commands

scontrol show partition -a | grep PartitionName=      # list partitions
sinfo -hNo '%f' | tr -d ' ' | tr ',' '\n' | sort -u   # list node features
sacctmgr show qos format=Name,GrpTRES                 # list QOS quotas

Development

pip install -e .
python -m pytest tests/ -q        # 84 tests, no SLURM required
python -m flake8 clust.py tests/