slurmscope

Interactive terminal dashboard for any Slurm cluster: Python >= 3.8, standard library only, no dependencies.

Install

pipx install slurmscope      # recommended: isolated, puts `slurmscope` on your PATH
pip install --user slurmscope
python -m slurmscope         # equivalent to the `slurmscope` command

It needs the Slurm client commands (squeue, scontrol) in PATH on the machine where you run it, typically a login node.

Version 1.4: one collector per login node serves every user's instances, nodeocc-style, with per-source intervals (see Cost and data). Version 1.2 added an AImageLab-HPC profile based on all six Using the Cluster documentation pages. Live Slurm configuration, squota, and susage remain the source of operational values.

slurmscope                 # UI every 1 s; queue every 5 s, nodes 30 s (shared per login node)
slurmscope -p <partition>  # start filtered on one partition
slurmscope -u <user>       # whose jobs are highlighted / listed in the overview (default: you)
slurmscope --page 4        # start on memory / storage
slurmscope --once          # single overview snapshot (pipe-friendly); --no-color, --ascii, --no-mouse
slurmscope --once --page 4 --scan-usage  # RAM, quota, and directory sizes

Views (1 2 3 4 / Tab, or click the tab bar)

• Overview - KPIs + history sparklines, per-partition load, GPU models (free GPUs), your jobs (approximate priority rank, wait, backfill start estimate), pending reasons, top users, and a QoS summary. The overview adapts to the terminal height (side-by-side panels from 150 columns, lists trimmed to fit); it scrolls only when the terminal is too small, and the footer then shows the current line range. Press Q or Enter to browse all configured QoS, including ones with no jobs. Each row shows running/pending experiments, distinct users, allocated GPUs/CPUs, and (on wider terminals) priority and maximum wall time. Search by name, description, or user; sort or scroll through the complete list. i shows resource/job limits, policy flags, user activity, and pending reasons. Enter on a QoS opens its experiments across all visible users, clearing job-specific user/state/search filters; Esc returns to the browser. The partition filter still applies to experiment counts/lists; P cycles back to all partitions. QoS details distinguish shared GPU caps from individual caps and calculate visible GPU headroom across all partitions. Partition policy adds wall-time bounds and billing weights; associations, partition QoS, access rules and flags can further change the effective limits.
• Jobs - every job in the queue, scrollable. Running jobs of everybody or only yours. QoS is searchable, sortable, shown in job details and, on wide terminals, in its own column. Project accounts are searchable/sortable, shown on very wide terminals, and filterable with a or -A PROJECT. v opens live sprio factors and the job's scontrol SiteFactor. A displayed priority rank is not a guaranteed execution order or start-time estimate.
• Nodes - every node: state, CPU / GPU / memory bars, load, partitions. v cycles minimum GPU VRAM; --min-vram GB sets it initially. Values come from live GPU feature tags, not hardware assumptions or GPU-memory-utilization probes. Exact feature tokens are searchable; node detail includes a quoted --constraint suggestion and raw GRES/tmpfs data.
• Memory - cluster configured/allocated RAM, capacity on nodes that are up and not drained, unallocated RAM on those nodes, OS-reported free RAM, allocations by user (selected user first), and RAM by node. Also includes the selected user's disk quota and on-demand directory sizes. The page is compact: users/nodes are packed into columns and trimmed to fit the terminal height; it scrolls only when the terminal is too small.

AImageLab-HPC integration

slurmscope --profile aimagelab
slurmscope --profile aimagelab --page 4
slurmscope --profile aimagelab --page 3 --min-vram 24 -p all_usr_prod
slurmscope --profile aimagelab --page 2 -A my_project

The default --profile auto recognizes an AImageLab/AILB cluster name, an ailb-* hostname, or the presence of both squota and susage. --profile generic explicitly disables site integration. You can force aimagelab when using a workstation with Slurm client tools.

• Storage: absolute /homes/USER, /work, and /dres paths; personal/project usage from squota; capacity rows for associated project paths. Project counters include collaborators. Recursive scans under /work and /dres, including aliases, are skipped in this profile.
• B — Budgets: all projects reported by susage, current-year remaining standard hours, low/exhausted balances, and original counters. The overview also shows a compact summary. Current-year columns are selected by their actual header; missing values stay unknown.
• v on a job — Priority: live component values, with the separate site factor. Recent-use fairness changes priority; exhausted annual budgets can reject paid submissions. This follows the newer Budget page where its annual-budget restriction qualifies the older Priority page.
• E — Session: loaded modules, local Python, job/session context and CUDA visibility. On ailb-login-02, an active CUDA 12+ module produces the documented CUDA 11.8 compatibility hint. No modules are loaded or unloaded, and another job's environment is never inferred.
• D — Guide: brief usage guidance, storage lifecycle, execution/transfer hints and links to every source page. The reference was reviewed on 2026-11-01; live data takes precedence.

squota and susage run as the logged-in Unix account. -u OTHER_USER only changes job highlighting and RAM summaries; it does not impersonate that user for these site commands. Their identity is labelled in the UI. Missing commands, changed table formats and query errors are shown explicitly; cached counters remain visible and marked stale after a failed refresh. No job submissions, module changes, transfers or filesystem writes are performed by this integration.

Documentation page	Integration
Environment and Customization	Session/module inspection and CUDA compatibility hint
Scheduler and Job Submission	Project accounts, QoS caps and partition policy
Node Features and Constraints	Minimum VRAM filters and constraint tokens
Budget and Accounting	Live project budgets and balance summaries
Job Priority	Priority components, SiteFactor and approximate-rank labels
File Systems and Data Management	Site quotas, project scope and BeeGFS scan guard

QoS visibility

QoS definitions come from a read-only sacctmgr show qos query, cached for 15 minutes. The experiment list uses jobs currently visible to squeue, not completed-job history. Pending array ranges count as one queue row, matching the existing Jobs view. Site privacy settings may hide other users' jobs. Definitions describe the accounting database's QoS; access to submit jobs under them can still depend on your associations and partition policies. If sacctmgr/SlurmDBD is unavailable or access is denied, a warning appears and QoS found in the queue remain browsable; cached definitions are retained on failure.

Memory and disk scope

RAM figures are Slurm allocations, not measured process RSS. Per-user RAM comes from allocated TRES for active jobs; pending requests are not counted. Missing allocation data (including the legacy squeue -o fallback) is marked unknown, not zero. Cluster allocation is read from nodes, so it can differ from the sum of visible jobs. OS free memory is informational, can be stale, and is reported with the number of nodes providing a value. Nodes are counted once even when they belong to several partitions.

Disk data describes filesystems mounted on the host running slurmscope. Slurm does not provide a cluster-wide filesystem inventory. By default the selected user's home and the working directory are reported. Add shared storage paths explicitly:

slurmscope --page 4 --storage-path /scratch --storage-path /projects
slurmscope -u alice --page 4 --usage-path /scratch/alice --usage-path /projects/alice

--storage-path adds filesystem-capacity rows. --usage-path selects directory scan roots (repeatable; replaces the default selected-user home). Filesystem used includes all users; available excludes reserved blocks. Paths on the same filesystem can show the same capacity, so those rows must not be added. The P filter affects RAM/jobs, not storage scope.

In the generic profile, user ownership and limits are shown using quota -w -u USER when available (the standard block unit is KiB). Quota support/permissions depend on the site; some shared filesystems require their own quota tools. If quota is unavailable, press d to measure your home or configured directories. Scans use du -skx, stay on one filesystem, run in the background, and time out after 180 seconds per path (a home on a network filesystem can take over 30 s); each path shows scanning Ns / queued and its result as soon as it is ready. A directory footprint includes files of all owners in that directory, and is not a complete ownership-based quota. Nested paths must not be summed. Permission failures and partial scans are labelled; previous samples remain visible during a rescan. Directory scans run only on demand, or with --once --page 4 --scan-usage.

Keys

key	action
↑ ↓ j k, PgUp PgDn, g G, wheel	move / page / top / bottom
Enter, double-click	details: job (resources, files, queue position, ...) or node (+ its jobs)
Q, Enter on overview	full QoS browser; Q / Esc returns to the overview
Enter, i in QoS browser	experiments of all visible users / QoS limits and user activity
4, d, r	memory/storage page / scan directories / refresh RAM and storage
B, E, D	project budgets / current session and modules / AImageLab guide
a, v in Jobs	selected project filter / priority components
v in Nodes	cycle minimum GPU VRAM filter
l	tail the job's stdout (follows; ↑ scrolls back, End follows again, e switches to stderr)
/	search (words are ANDed: bob a100 pending)
m u s f	only mine / only the selected user / cycle state / GPU jobs (nodes: free GPUs)
o O, click header	next sort column / reverse / sort by the clicked column
P	cycle partition filter (all views)
c	cancel the selected job - own jobs only, validated job id, asks y/N
x	clear filters
p/Space, + -	pause, refresh interval (0.5-60 s)
?, q	scrollable help, quit (closes the current screen first)

Mouse capture hides terminal text selection; hold Shift to select, or use --no-mouse.

Cost and data

One master per login node, shared by every user (like nodeocc). On each login node, one slurmscope process (the master) queries Slurm; every other slurmscope of every user on that node is a client and only reads the master's snapshots. One login node has one master; different login nodes have different masters (there is no cluster-wide collector). No root, service account or administrator action is needed.

slurmscope                    # joins this node's master, or becomes it (an interactive UI can be master)
slurmscope --master-only      # optional: run just the collector (tmux/nohup), no UI
slurmscope --personal         # explicit opt-out: coordinate only your own instances

Election. The master owns a Linux abstract Unix socket whose name derives from the Slurm configuration (SLURM_CONF, SLURM_CONF_SERVER, SLURM_CLUSTERS). Binding it is atomic and common to all UIDs; there is no lock or socket file to delete, replace or leave behind. If the master dies, any instance (of any user) takes over at its next refresh. Slurm commands inherit the listening socket, so the name stays taken while an orphaned query of a dead master is still running: a successor never overlaps with it. If coordination fails, slurmscope shows an error and never starts independent queries; personal mode is never a silent fallback.

Trust model (accepted explicitly, as for nodeocc). Any local user may become master and any local user may connect. A malicious local user can therefore win the election or squat the name, serve false data or block the service. SO_PEERCRED only reports which UID is serving (it is not proof that the master is trustworthy). Data received from a master is sanitized like any external text before it reaches the terminal. Use --personal if this is not acceptable.

Intervals, decided by the master. Each source has its own interval and its own worker:

Source	Default	Option
UI refresh (local, no Slurm)	1 s	-i
main squeue (queue)	5 s	--jobs-interval (5–60; 1–4 only with --allow-fast-squeue)
scontrol show node, squeue --start (only if jobs are pending)	30 s	--share-interval (30–60)
scontrol show config (PrivateData check), QoS, partitions	15 min	–
squota, susage	1 h	always personal, private cache

There is at most one query in flight per source; the next run is scheduled from the end of the previous one, so a slow query delays its own source without accumulating requests or bursting to catch up. After an error or timeout the source backs off (2×, 4×, 8× its interval, up to 5 minutes) and keeps its previous data. The status line shows the mode, the role and the real age of each sample (host client · jobs 3s nodes 12s starts 12s; ! marks a failing source); errors are listed per source with data age and retry time. Clients cannot request collections or change intervals: options such as --jobs-interval only matter in the process that is (or becomes) master, so after a takeover the new master's options apply.

--allow-fast-squeue --jobs-interval 1 queries the whole queue every second from each login node's master: this increases the load on slurmctld five-fold compared with the default and gives no guarantee of respecting your cluster's usage policy; ask your administrators first.

Compatibility. --share-interval used to set the single collection cycle for everything (30–60 s). It keeps the same range and now sets the node-state and start-estimate interval; the queue uses --jobs-interval (add --jobs-interval 30 for the old queue cadence). The previous fixed-account shared mode (--shared-socket, --master-user, coordinator.conf) is replaced by this host mode and its options were removed.

Cache. Each sample is kept with its own timestamps and failure count, also in a persistent cache: in host mode one file per writer UID in /tmp (slurmscope-HOST-NS-uUID.snapshot, mode 0644; a process never modifies or deletes another user's file and refuses symlinks, FIFOs and files whose owner differs from the UID in their name), merged per source when read. Restarts, takeovers by another user and --once reuse valid samples and query only the sources that are due; an instance uses the cache without becoming master only when no source is due. SLURMSCOPE_HOST_CACHE_DIR moves these files (tests use it).

Privacy. The legitimate master reads Slurm's PrivateData every 15 minutes; if it includes jobs, all, nodes or partitions, or cannot be verified, it queries and publishes nothing but an error, and clients show it without falling back. Use --personal on such clusters.

Personal mode (--personal) keeps the private model: a path socket in a 0700 directory ($XDG_RUNTIME_DIR/slurmscope or ~/.cache/slurmscope, or --cache-dir), an flock election limited to your UID and SO_PEERCRED checked in both directions. Missing directories are created without following symlinks, with modes independent of the umask.

The selected job's scontrol show job (log paths, details) and on-demand sprio remain personal queries, outside the shared collector.

Command arguments. Values that come from a master or a cache are never trusted as command-line arguments. Before a cancel confirmation is shown, and again right before scancel runs, the job id must match the formats squeue prints (123, 123_4, 123_[1-5,7%2], 123+0); anything else (--user=…, --me, empty values, spaces, non-ASCII digits…) is refused with an error and no command runs. The same check applies to scontrol show job, sprio and node/partition names, and scancel/scontrol receive the id after the end-of-options marker --. Terminal sanitization is a separate protection and does not replace this validation.

Personal quota and budget data. squota, susage, and job detail snapshots remain in a private per-user local cache (0700, --cache-dir), never in shared snapshots. In the AImageLab interactive profile, quota and budget load at startup in the background and are checked hourly, reusing samples younger than one hour across program restarts. Generic quota loads on page 4 and is checked hourly. Explicit refresh does not bypass personal snapshot lifetimes. Personal cache/lock failures do not trigger uncoordinated fallback queries. A priority report is queried on demand and cached in the process for 15 minutes. Only one background query/scan of each kind can be in flight per process. Site quota/budget tables retain raw output when their columns cannot be recognized; optional tools are not required for core monitoring. GPU counts come from the allocated TRES, so GPUs assigned by site defaults are counted too (falls back to squeue -o %b on Slurm versions without tres-alloc).

Implementation references: squeue, QoS fields, and scontrol.

Development

git clone https://github.com/FabioBozzoli/slurmscope && cd slurmscope
pip install -e ".[test]"
python -m pytest

Layout: src/slurmscope/cli.py (dashboard and CLI), tests/test_slurmscope.py. Released under the MIT license (see LICENSE).