lazyslurm-py 0.3.0

Displays the tail of the job's standard output and error log files. LazySlurm finds log files by reading StdOut / StdErr from scontrol show job. For older jobs no longer in Slurm's memory, it checks the log path cache first (see Log Path Cache), then falls back to searching the working directory for common patterns (slurm-JOBID.out, JOBNAME-JOBID.out, logs/ subdirectory, etc).

Live process listing from the job's compute node, similar to top. Shows PID, %CPU, %MEM, RSS, VSZ, elapsed time, and command name. Auto-refreshes while the tab is active.

Live nvidia-smi output showing only the GPUs allocated to the selected job. Uses srun --overlap --jobid to run nvidia-smi inside the job's cgroup, so GPU visibility is automatically restricted to the job's allocation. The header shows CUDA_VISIBLE_DEVICES for confirmation. Auto-refreshes while the tab is active.

Accounting statistics from sstat (running jobs) and sacct, starting with what the job used against what it reserved:

Efficiency
  CPU            <0.1 / 8 cores    <1% ▁▁▁▁▁▁▁▁  ← over-requested
  Memory         2.6G / 64G         4% ▁▁▁▁▁▁▁▁  ← over-requested
  GPU               1 / 1 allocated      — utilisation is not recorded by Slurm
  Walltime      17:02 / 12:00:00    2% ▁▁▁▁▁▁▁▁
  next time try --mem=4G --cpus-per-task=1 --time=00:25:00

• CPU is TotalCPU / (cores × elapsed), the same definition seff uses
• Memory compares the peak RSS of one task against the request per node, so a multi-node job is not credited with memory it never touched on any single node. At 100% or above the row turns red — the next run risks being killed for exceeding its request
• Walltime is elapsed / time limit
• Green above 60%, yellow 25-60%, red below. A fraction of a percent shows as <1% rather than rounding up to 1%
• The last line suggests a request about a third larger than what was actually used, and only appears when something was clearly over-provisioned
• Array tasks are measured per task; a job too old for sacct reads unavailable

Then the raw accounting fields:

• CPU — average CPU time, total CPU, frequency, wall time
• Memory — requested, max/average RSS, max/average VM size, peak node/task
• GPU — allocated GPU count and type from TRES
• Disk I/O — average and max read/write

Partition, node count, CPUs, memory, GPU/GRES allocation, TRES, time limit, runtime, account, and QoS.

Submit time, start/end times, working directory, stdout/stderr paths, and the original submit command.

All key-value pairs from scontrol show job or sacct, displayed verbatim.

Job names and partition names are truncated to 16 characters by default (with … when truncated). This keeps the tables compact. Configure via config.toml:

max_name_width = 20       # wider name column
max_partition_width = 10  # narrower partition column

Set to 0 for unlimited width.

For compact displays, enable abbreviated state names in the Terminated Jobs table:

abbreviate_states = true

Each partition is assigned a consistent color across both job tables. Colors are deterministic (based on the partition name) so they stay stable across sessions. You can override colors in the config file (see Configuration).

Active Jobs (Job ID column):

Terminated Jobs (State column):

The top line shows a summary of your jobs and cluster partitions:

jdoe  5 running  2 pending    gpu:10/5/1/16  cpu:42/58/0/100

Partition format is name:A/I/O/T:

/ opens the filter bar. Plain words search the job id, name and partition (and state, in the Terminated table) as before, and key:value terms narrow by field. Terms are ANDed.

Enter accepts the filter: the bar closes, the query stays in force, and the cursor lands back on the matching rows so you can cancel, edit or inspect them. Escape abandons it, clearing the query and showing every job again. While a filter is active the table's border title says so — Active Jobs — 2/4 match.

state:pend part:gpu        pending jobs on the gpu partition
train state:run            running jobs whose row mentions "train"
gpu:>0                     jobs that asked for at least one GPU
name:sweep id:4815         both must match

Quote values containing spaces: name:"my long job". An unknown key is treated as plain text — foo:bar just searches for the string foo:bar, so nothing you type can break the filter. The panel border shows how much matched (Active Jobs — 2/4 match), and a filter that matches nothing says no jobs match instead of showing an empty table.

gpu: only applies to the Active table: sacct rows carry no GRES, so the term matches nothing among terminated jobs.

A 40-task array would otherwise fill the table with 40 near-identical rows, so tasks of one array are folded into a single row:

▸ 4815201_[0-11] ×12   sweep-lr   2run 10pend   gpu

The Job ID cell shows the task-index range and the total task count; ×12 counts tasks, not rows, so a pending 4815201_[3-11] block counts as the nine tasks it stands for. The Elapsed column carries a state tally instead of a time, which means little across a dozen tasks that started at different moments. In the Terminated table the tally sits in the State column and Elapsed shows the longest run of the array.

Enter expands the row into its tasks (├/└ prefixed) and collapses it again. Expansion survives refreshes and filter changes, so a group you opened stays open.

Actions on a collapsed row act on the whole array: c cancels it with a single scancel <base id>, u edits every pending task in it, m bookmarks the group so it pins to the top as a unit. On an expanded task, everything behaves per task as usual. The detail panels always show a real task (the first one) — Slurm cannot describe a bare base id.

Turn it off with collapse_arrays = false in config.toml to get one row per task again.

Press m to bookmark any job. Bookmarked jobs are pinned to the top of their table with a ★ prefix. Bookmarks persist for the duration of the session.

Press Shift+U for what the allocation has cost so far and what it is doing to your priority:

this month   8 200 CPU-hours used by you   account total 12 500  66% of it yours   (w cycles window)
+------------------------------- Fair share -------------------------------+
| physics  factor 0.437  entitled 1.68%  used 5.64%                        |
|   over your share — your jobs get reduced priority (using 3.3x your share)|
+------------------------------ Account usage -----------------------------+
| User      Name        CPU hours  Share              %                    |
| ▸ jdoe    Jane Doe        8 200  ████████████░░░░  66.1%                 |
|   asmith  A Smith        3 100   ████░░░░░░░░░░░░  25.0%                 |
|   bpatel  B Patel        1 200   ██░░░░░░░░░░░░░░   9.7%                 |
+--------------------------------------------------------------------------+

• Hours come from sreport cluster AccountUtilizationByUser, per user in your account, biggest consumer first, your own row marked ▸.
• Fair share comes from sshare — this is what actually drives queue priority. entitled is your slice of the cluster, used is the slice you have consumed, and the factor is Slurm's verdict: above 0.5 you are under-consuming and get boosted, below it you are over-consuming and get pushed back. The sentence underneath says which.
• w cycles the window: this month → last 30 days → this year. r refetches, Escape / Shift+U / q returns.

sreport can take seconds on a busy accounting database, so the screen opens immediately with loading usage... and fills in when the data arrives. Nothing here runs in the poll loop — it is fetched on open, on r, and when the window changes. On a cluster without Slurm accounting the panel says so rather than showing an empty table.

Press p for a full-screen view of the cluster's partitions. The main job tables only ever show your jobs — this screen shows everyone's, so you can see what a partition is actually busy with before you queue into it.

Partition table — A/I/O/T is Slurm's allocated / idle / other / total counter, given for both nodes and CPUs. "Other" is down, drained, or reserved capacity. The Load bar is allocated CPUs over usable CPUs (allocated + idle), so drained nodes don't make a saturated partition look half empty; it turns yellow past 60% and red past 90%. Run and Pend are job counts across all users, and a pending job that names several partitions counts in each. Partitions that are down are shown struck-through instead of with a bar, rather than hidden.

Job table — every user's jobs on the highlighted partition, running first then pending, newest first. Your own jobs are marked ▸ and highlighted. For pending jobs the last column shows Slurm's reason ((Resources), (QOSMaxGRESPerUser), (Dependency), …) instead of a node list.

The screen re-polls on your --refresh interval while it is open, and stops when you leave. --partition-order also orders this table. Note that sinfo --summarize reports one row per node configuration, so partitions with mixed hardware are summed into a single row here.

Press Enter on a partition to see its individual nodes:

Nodes without GRES show — in the GPU column, and a drained node's load is shown as — because its counters say nothing useful — the reason does.

The bar at the top counts the partition's nodes by state and totals GPUs in use. The lower panel lists all users' jobs running on the highlighted node, so you can see who you would be sharing it with. Up/Down moves between nodes and the job list follows, Tab switches panels, r refreshes, Escape or q goes back to the partition list.

Older Slurm versions that do not support the GresUsed output field fall back automatically to a shorter query; everything except the GPU column still works.

Press u on a pending job to open the property editor:

The modal reads like a small file you edit: numbered lines, the scontrol key as the label, the job id as the "filename" in the border. The values are prefilled from the job. Up/Down and Tab/Shift+Tab move between lines (wrapping at either end), Left/Right, Home/End and Backspace edit within a line, Ctrl+S writes, Escape quits. Only fields you actually changed are sent, as a single scontrol update jobid=<id> Key=Value ... per job, and every command plus Slurm's reply is written to the Command Log.

Pending jobs only. Once a job starts, Slurm fixes its runtime, partition, and resource allocation, so the editor refuses to open for running jobs and says so in the status line.

With a Ctrl+V multi-selection active, u edits all selected pending jobs at once. Then the fields start blank and only the ones you fill in are applied to every job — non-pending jobs in the selection are skipped and listed in the Command Log.

Press b to open the selected job's sbatch script in your editor, read-only. Works on pending, running, and terminated jobs. The TUI suspends while the editor is open.

With vim the file opens via -R, so you can still :w /some/other/path to keep a copy — useful when the original script has been changed since the job was submitted. Editors without a known read-only flag open writable, and the Command Log says so.

The script text is archived, not just its path, so your copy stays readable even if you later edit, move, or delete the original file. Archiving happens automatically whenever you select a job that Slurm still knows about, and on demand when you press b. Array tasks share one script: 123_11, 123_[1-40], and 123 all resolve to the same file.

Slurm only keeps a job's script until MinJobAge seconds after it ends — often just 300. A job that finished before LazySlurm ever saw it has no archived copy and Slurm can no longer produce one; pressing b reports that the script is unavailable. See Job Cache for the cache location and script_cache_dir.

The stdout/stderr tabs show the last 500 lines, read by seeking backwards from the end of the file — a 200 MB log tails as fast as an empty one, and the panel never blocks the UI while a job floods its log.

To read more than the tail, press l to open the active tab's log in a pager (less by default). This is the right tool for a multi-gigabyte log: it seeks instead of loading, so it opens instantly at the end of the file, and inside it you get

The TUI suspends while the pager is open. In remote mode the pager runs on the cluster over the existing SSH connection, so nothing is copied to your machine and you are not asked for a 2FA code again.

Set your pager in config.toml:

pager = "less"   # or "more", "bat", "most"

less is opened with -R +G (keep the log's colors, start at the end); other known pagers get their equivalent end-of-file flag. An unknown pager is run with no flags.

If a log has no line break in its last 4 MB — a progress bar writing \r forever — the tail is cut there and the panel says so; press l to see the file properly.

Press e to open the selected job's stdout log in an external text editor, or Shift+E for stderr. The TUI suspends while the editor is open and resumes when you close it.

The default editor is vim. To change it, set editor in your config file:

editor = "nano"    # or "vim", "less", "code", etc.

In remote mode, the log file is first copied to a local temp file via scp, opened in the editor, and cleaned up when the editor closes.

If the configured editor is not found on your system, an error is shown in the Command Log (e.g., editor 'code' not found — set 'editor' in config.toml).

Press o to get a shell on the compute node running the selected job. There are two ways to do that, and they land you in genuinely different places:

interactive_shell = "ssh"    # "ssh" (default) | "srun"

ssh — the machine. You get a normal login shell on the node, outside the job's cgroup. CUDA_VISIBLE_DEVICES is unset, nvidia-smi shows every GPU on the node rather than the one you allocated, and anything you start there is not capped by the job's CPU or memory limits — so a heavy process competes with your job instead of being contained by it. In exchange it adds no job step, so it cannot skew the efficiency report, and it either connects or fails fast.

srun — the allocation. LazySlurm runs srun --overlap --jobid=<id> --pty bash, so the shell lands inside the job's cgroup: correct CUDA_VISIBLE_DEVICES, correct resource limits, the same environment the job sees. The costs are real, though: the shell appears in sacct as a job step, and an idle debugging shell drags the job's reported CPU efficiency down. It also needs Slurm ≥ 20.11 for --overlap, and can block while negotiating the step launch.

So: use ssh to poke at the machine, srun to debug inside your allocation. If nvidia-smi shows you more GPUs than you asked for, that is the ssh path working as designed — switch to srun for that shell.

You do not have to choose once and live with it: o uses the configured method and Shift+O uses the other one, for a single shell.

Two cases where you may have to use srun:

• Your cluster runs pam_slurm_adm, which refuses SSH to a compute node without an allocation there.
• You are reproducing something that depends on the job's environment or limits.

srun needs a live allocation, so it only applies to a running job. On anything else LazySlurm falls back to ssh and says so in the Command Log. If the step launch fails, it reports the exit status rather than silently connecting you somewhere else.

When a running job finishes (completes, fails, times out, etc.), LazySlurm:

• Rings the terminal bell
• Attempts a desktop notification via notify-send (Linux)
• Logs the event in the Command Log panel

The bottom-right panel shows a timestamped log of all actions and their results:

14:23:05 refresh
  >>> complete
14:23:12 scancel 2465400
  >>> Job 2465400 cancelled.
14:23:30 ssh galvani-cn109
  >>> session to galvani-cn109 closed
14:24:01 job completed
  >>> 2465485 COMPLETED

Archived as text under the base job ID, so all tasks of an array share one file. See View sbatch Script for the b keybinding and its limitations.

Log paths are cached the same way, into log_cache.json, whenever you select a job that Slurm still knows about. For older jobs LazySlurm falls back to guessing from filename patterns (slurm-JOBID.out, JOBNAME-JOBID.out, logs/ subdirectories), which can fail if you use custom --output/--error names.

Resubmit (s) runs the job's original sbatch command. If the script file it names no longer exists, LazySlurm substitutes the archived copy and says so in the Command Log. Not available in remote mode, where the archive is local but sbatch runs on the login node.

The loop after a failure is usually "run it again, but bigger" — more time after a TIMEOUT, more memory after an OOM kill. u cannot help there: Slurm fixes a job's allocation once it starts, so the property editor only works on jobs still queued.

Shift+S opens that same editor for a terminated job, prefilled with what the job actually had, and submits with the changed fields as sbatch flags:

sbatch --chdir /work --time=4:00:00 --mem=16G job.sh

• Fields map to --time, --partition, --nodes, --cpus-per-task and --mem.
• An override replaces the same option in the original submit line rather than being appended next to it, so the command log shows exactly what was requested.
• A field left blank keeps whatever the original line had. Options after the script name belong to the script and are never touched.
• After a TIMEOUT the runtime field is prefilled with double the old limit, and after OUT_OF_MEMORY the memory field with double the old request. They are suggestions — overwrite or clear them.
• The full sbatch line is written to the Command Log before it runs.

The archived-script fallback applies here too.

Both are pruned on startup using cache_max_age_days (default 30, 0 to never prune). Set script_cache_dir in config.toml to archive scripts somewhere else.

Earlier versions shipped a lazyslurm-daemon that polled for log paths in the background. It has been removed — caching now happens inline. A leftover ~/.config/lazyslurm/daemon.pid is inert and can be deleted.

LazySlurm opens a single SSH session in the background at startup and runs everything through it. It does not spawn ssh per Slurm call:

1. An SSH master connection is started on a pty and authenticated once.
2. A shell channel (/bin/sh -s) is opened over that master and kept alive for the whole session.
3. Each squeue, sacct, scontrol, sstat, scancel, sbatch — and every log-file read — is written into that shell and its output read back. No new connection, no new process, no re-authentication.

Commands are serialized on the channel, so they queue rather than interleave. If the channel dies (network drop, remote logout), the next command reopens it automatically, and only re-authenticates if the master itself is gone. The session is closed when you quit.

Because the master owns a pty, whatever the cluster asks at login is captured and shown to you in a modal instead of being lost:

The label is the server's own prompt text, so it reads exactly as it would in a terminal — Password:, Verification code:, Passcode or option (1-3):, Token_Response: and so on. Input is masked (host-key (yes/no) questions are not). You are asked once, at startup, because every later command reuses that connection.

If you mistype a code or press Escape, the Command Log says so and nothing is polled; press r to retry the connection. Failures report the server's own message (for example Permission denied (keyboard-interactive)) rather than a generic timeout.

Everything else that shells out reuses the same authenticated connection, so 2FA is never requested twice:

The control socket lives in ~/.ssh/cm-lazyslurm/. If you already have a master connection to that host, it is reused and you are not prompted at all — passwordless keys therefore still connect with no interaction.

When using --remote user@host, the username is automatically used as the default --user for Slurm queries (no need to specify both).

The archived-script fallback for resubmission is local-only: the archive lives on your machine while sbatch runs on the login node.

Login node warning: If the local or remote hostname contains "login", LazySlurm shows a warning popup reminding you to be mindful of resource usage on shared login nodes.

# All CLI arguments can be set here as defaults.
# CLI arguments always override config file values.
# When a CLI arg overrides a config value, it is shown in the Command Log.

refresh = 3.0            # -r/--refresh: auto-refresh interval in seconds (0 = off)
days = 14                # -d/--days: how many days back for terminated jobs
user = "myuser"          # -u/--user: Slurm user to monitor
partition = ""           # -p/--partition: filter by partition (empty = all)
no_gpu = false           # --no-gpu: disable GPU monitoring tab
no_live = false          # --no-live: disable live CPU/GPU monitoring
remote = ""              # -H/--remote: SSH target for remote mode
editor = "vim"           # text editor for viewing logs ("vim", "nano", "less", etc.)
pager = "less"           # pager for browsing whole logs with 'l' ("less", "more", "bat")

# Column display settings
max_name_width = 16      # max characters for job name column (0 = unlimited)
max_partition_width = 16 # max characters for partition column (0 = unlimited)
abbreviate_states = false # use short state names: COMP, FAIL, TIME, CAN, OOM, ...
collapse_arrays = true   # fold a job array into one row; Enter expands it

# Cache settings
# cache_max_age_days = 30  # auto-delete cached job info older than N days
                           # set to null to never delete (keep forever)
# script_cache_dir = ""    # where archived sbatch scripts live
                           # (default: ~/.config/lazyslurm/scripts)

# Partition display order in the cluster bar.
# Partitions not listed appear after these in their default order.
# Set via CLI: lazyslurm --partition-order gpu,cpu,fat
partition_order = ["gpu", "cpu", "fat"]

# Custom partition colors in the job tables.
# Overrides the automatic hash-based coloring.
# Valid color names: cyan, magenta, yellow, green, blue, red,
# bright_cyan, bright_magenta, bright_green, white, dim, bold,
# or any Rich color (e.g. "dark_orange", "grey50").
[partition_colors]
gpu = "green"
cpu = "cyan"
fat = "magenta"
debug = "dim"

All CLI arguments can be set in the config file. The precedence is:

CLI argument > config file > built-in default

When a CLI argument overrides a config file value that differs, the override is logged in the Command Log panel at startup.

To set a custom partition order for the cluster bar:

# Set once — automatically saved for future sessions
lazyslurm --partition-order gpu,cpu,fat

lazyslurm [-h] [-r SEC] [-d N] [-u USER] [-p PARTITION]
         [--no-gpu] [--no-live] [--partition-order P1,P2,...] [-H HOST]