slurmscope

Interactive terminal dashboard for any Slurm cluster: Python >= 3.8, standard library only, no dependencies.

Install

pipx install slurmscope      # recommended: isolated, puts `slurmscope` on your PATH
pip install --user slurmscope
python -m slurmscope         # equivalent to the `slurmscope` command

It needs the Slurm client commands (squeue, scontrol) in PATH on the machine where you run it, typically a login node.

Version 1.2 adds an AImageLab-HPC profile based on all six Using the Cluster documentation pages. Live Slurm configuration, squota, and susage remain the source of operational values.

slurmscope                 # live, refresh every 1 s
slurmscope -p <partition>  # start filtered on one partition
slurmscope -u <user>       # whose jobs are highlighted / listed in the overview (default: you)
slurmscope --page 4        # start on memory / storage
slurmscope --once          # single overview snapshot (pipe-friendly); --no-color, --ascii, --no-mouse
slurmscope --once --page 4 --scan-usage  # RAM, quota, and directory sizes

Views (1 2 3 4 / Tab, or click the tab bar)

• Overview - KPIs + history sparklines, per-partition load, GPU models (free GPUs), your jobs (approximate priority rank, wait, backfill start estimate), pending reasons, top users, and a QoS summary. The overview adapts to the terminal height (side-by-side panels from 150 columns, lists trimmed to fit); it scrolls only when the terminal is too small, and the footer then shows the current line range. Press Q or Enter to browse all configured QoS, including ones with no jobs. Each row shows running/pending experiments, distinct users, allocated GPUs/CPUs, and (on wider terminals) priority and maximum wall time. Search by name, description, or user; sort or scroll through the complete list. i shows resource/job limits, policy flags, user activity, and pending reasons. Enter on a QoS opens its experiments across all visible users, clearing job-specific user/state/search filters; Esc returns to the browser. The partition filter still applies to experiment counts/lists; P cycles back to all partitions. QoS details distinguish shared GPU caps from individual caps and calculate visible GPU headroom across all partitions. Partition policy adds wall-time bounds and billing weights; associations, partition QoS, access rules and flags can further change the effective limits.
• Jobs - every job in the queue, scrollable. Running jobs of everybody or only yours. QoS is searchable, sortable, shown in job details and, on wide terminals, in its own column. Project accounts are searchable/sortable, shown on very wide terminals, and filterable with a or -A PROJECT. v opens live sprio factors and the job's scontrol SiteFactor. A displayed priority rank is not a guaranteed execution order or start-time estimate.
• Nodes - every node: state, CPU / GPU / memory bars, load, partitions. v cycles minimum GPU VRAM; --min-vram GB sets it initially. Values come from live GPU feature tags, not hardware assumptions or GPU-memory-utilization probes. Exact feature tokens are searchable; node detail includes a quoted --constraint suggestion and raw GRES/tmpfs data.
• Memory - cluster configured/allocated RAM, capacity on nodes that are up and not drained, unallocated RAM on those nodes, OS-reported free RAM, allocations by user (selected user first), and RAM by node. Also includes the selected user's disk quota and on-demand directory sizes. The page is compact: users/nodes are packed into columns and trimmed to fit the terminal height; it scrolls only when the terminal is too small.

AImageLab-HPC integration

slurmscope --profile aimagelab
slurmscope --profile aimagelab --page 4
slurmscope --profile aimagelab --page 3 --min-vram 24 -p all_usr_prod
slurmscope --profile aimagelab --page 2 -A my_project

The default --profile auto recognizes an AImageLab/AILB cluster name, an ailb-* hostname, or the presence of both squota and susage. --profile generic explicitly disables site integration. You can force aimagelab when using a workstation with Slurm client tools.

• Storage: absolute /homes/USER, /work, and /dres paths; personal/project usage from squota; capacity rows for associated project paths. Project counters include collaborators. Recursive scans under /work and /dres, including aliases, are skipped in this profile.
• B — Budgets: all projects reported by susage, current-year remaining standard hours, low/exhausted balances, and original counters. The overview also shows a compact summary. Current-year columns are selected by their actual header; missing values stay unknown.
• v on a job — Priority: live component values, with the separate site factor. Recent-use fairness changes priority; exhausted annual budgets can reject paid submissions. This follows the newer Budget page where its annual-budget restriction qualifies the older Priority page.
• E — Session: loaded modules, local Python, job/session context and CUDA visibility. On ailb-login-02, an active CUDA 12+ module produces the documented CUDA 11.8 compatibility hint. No modules are loaded or unloaded, and another job's environment is never inferred.
• D — Guide: brief usage guidance, storage lifecycle, execution/transfer hints and links to every source page. The reference was reviewed on 2026-11-01; live data takes precedence.

squota and susage run as the logged-in Unix account. -u OTHER_USER only changes job highlighting and RAM summaries; it does not impersonate that user for these site commands. Their identity is labelled in the UI. Missing commands, changed table formats and query errors are shown explicitly; cached counters remain visible and marked stale after a failed refresh. No job submissions, module changes, transfers or filesystem writes are performed by this integration.

QoS visibility

QoS definitions come from a read-only sacctmgr show qos query, cached for 60 seconds. The experiment list uses jobs currently visible to squeue, not completed-job history. Pending array ranges count as one queue row, matching the existing Jobs view. Site privacy settings may hide other users' jobs. Definitions describe the accounting database's QoS; access to submit jobs under them can still depend on your associations and partition policies. If sacctmgr/SlurmDBD is unavailable or access is denied, a warning appears and QoS found in the queue remain browsable; cached definitions are retained on failure.

Memory and disk scope

RAM figures are Slurm allocations, not measured process RSS. Per-user RAM comes from allocated TRES for active jobs; pending requests are not counted. Missing allocation data (including the legacy squeue -o fallback) is marked unknown, not zero. Cluster allocation is read from nodes, so it can differ from the sum of visible jobs. OS free memory is informational, can be stale, and is reported with the number of nodes providing a value. Nodes are counted once even when they belong to several partitions.

Disk data describes filesystems mounted on the host running slurmscope. Slurm does not provide a cluster-wide filesystem inventory. By default the selected user's home and the working directory are reported. Add shared storage paths explicitly:

slurmscope --page 4 --storage-path /scratch --storage-path /projects
slurmscope -u alice --page 4 --usage-path /scratch/alice --usage-path /projects/alice

--storage-path adds filesystem-capacity rows. --usage-path selects directory scan roots (repeatable; replaces the default selected-user home). Filesystem used includes all users; available excludes reserved blocks. Paths on the same filesystem can show the same capacity, so those rows must not be added. The P filter affects RAM/jobs, not storage scope.

In the generic profile, user ownership and limits are shown using quota -w -u USER when available (the standard block unit is KiB). Quota support/permissions depend on the site; some shared filesystems require their own quota tools. If quota is unavailable, press d to measure your home or configured directories. Scans use du -skx, stay on one filesystem, run in the background, and time out after 180 seconds per path (a home on a network filesystem can take over 30 s); each path shows scanning Ns / queued and its result as soon as it is ready. A directory footprint includes files of all owners in that directory, and is not a complete ownership-based quota. Nested paths must not be summed. Permission failures and partial scans are labelled; previous samples remain visible during a rescan. Directory scans run only on demand, or with --once --page 4 --scan-usage.

Keys

Mouse capture hides terminal text selection; hold Shift to select, or use --no-mouse.

Cost and data

Per refresh: one squeue -O ...tres-alloc... and one scontrol show node (plus squeue --start every 15 s if you have pending jobs; scontrol show job only while a detail/log screen is open). QoS definitions refresh at most once per 60 s in a background worker. While page 4 is active, user quota also refresh at most once per 60 s in the background. Partition policies are cached for 60 s (scontrol show partition -o). The AImageLab profile also refreshes susage at most once per 60 s. A priority report queries only the selected job on demand, caches the result for 60 s, and supports explicit r refresh. Only one query/scan of each kind can be in flight. Optional failures never stop queue polling. --once --page 1 queries definitions once; --once --page 4 queries storage once. The AImageLab overview snapshot also queries its project budget once. Site quota/budget tables retain the original output when their column layout cannot be recognized; optional tools are not required for core queue/node monitoring. GPU counts come from the allocated TRES, so GPUs assigned by site defaults are counted too (falls back to squeue -o %b on Slurm versions without tres-alloc).

Implementation references: squeue, QoS fields, and scontrol.

Development

git clone https://github.com/FabioBozzoli/slurmscope && cd slurmscope
pip install -e ".[test]"
python -m pytest

Layout: src/slurmscope/cli.py (dashboard and CLI), tests/test_slurmscope.py. Released under the MIT license (see LICENSE).