slin 0.2.0

A friendly layer between you and the SLURM Workload Manager: a Rich-powered CLI, a local SQLite history of jobs and cluster resources, and a Textual TUI for live monitoring and job control.

slin summary        # the whole cluster at a glance
slin myjobs         # your queue
slin nodes --gpu    # who is eating the GPUs
slin tui            # the interactive monitor

Why it exists: squeue/sinfo/sacct print everything you need and nothing you can read. slin keeps the same data, adds colour-coded states and usage bars, and — because cluster accounting is usually purged after a few weeks — stores your job history locally so it survives that retention window.

• CLI dashboard — jobs, nodes, partitions, CPU/memory/GPU utilisation, fairshare.
• Local history — SQLite database of jobs, job steps and resource time series.
• TUI — tabbed live monitor with cancel/hold/release, per-node gauges and sparklines.
• Scriptable — --json on every command emits the same models as structured data.
• Light dependencies — typer, rich, textual. No ORM, no compiled extensions; it installs on a locked-down login node.

slin never submits jobs: it reads the cluster and controls jobs you already own.

---

Requirements

• Python 3.10+
• The SLURM client commands on PATH: squeue, sinfo, sacct, scontrol, sshare, scancel — i.e. run it on a login node, or anywhere squeue works.

Missing commands degrade gracefully: no sacct means no history sync, no sshare means slin fair says so instead of crashing.

Install

# from a checkout
pip install .

# or straight from GitHub
pip install git+https://github.com/gabrielvpina/slin

# isolated, recommended on shared login nodes
pipx install git+https://github.com/gabrielvpina/slin

Then check it:

slin --version
slin --help

python -m slin works too, if the console script is not on your PATH.

---

Quick start

slin summary                  # jobs, CPU, memory, GPU and partitions in one screen
slin myjobs                   # your jobs in the queue
slin details 4815162          # one job, section by section, with per-step usage
slin watch -v jobs -i 10      # live view, refreshed every 10s (Ctrl+C to stop)

slin sync --since 30d         # import the last 30 days of accounting into the DB
slin history --me             # read that history back, paged and filterable
slin stats --since 90d        # CPU-hours, GPU-hours, success rate per user/partition

slin tui                      # interactive monitor

---

Syntax

slin [GLOBAL OPTIONS] COMMAND [ARGS] [OPTIONS]

Global options

Global options go before the command.

Option	Meaning
-c, --config PATH	Config file (default $XDG_CONFIG_HOME/slin/config.toml)
--scope auto|all|user	Which users' data to query — see Scope
--db PATH	Override the history database path
--timeout SECONDS	Per-SLURM-command timeout (default 30)
--json	Emit JSON instead of tables
--no-color	Disable colour
--width N	Force output width (useful when piping)
-V, --version	Print the version and exit
-h, --help	Help for slin or for any subcommand

slin --json jobs -u alice          # correct
slin jobs --json -u alice          # wrong: --json is a global option

Filters shared by the view commands

Option	Meaning
-u, --user NAME	Filter by username
-p, --partition NAME[,NAME]	Filter by partition (comma-separated)
-S, --state STATE[,STATE]	Filter by state (comma-separated)

Time windows

Anywhere a --since / --keep window is accepted: 30d, 12h, 90m, 2w. A bare number means days — --since 7 is --since 7d.

---

Commands

Viewing the cluster

Command	What it shows
slin summary	Job states, per-partition CPU/mem/GPU utilisation, queue depth
slin jobs [-u USER] [-p PART] [-S STATE] [-n N]	The live queue
slin myjobs [-p PART] [-S STATE]	Your own jobs
slin nodes [-p PART] [-S STATE] [--gpu] [--no-detail]	Node table with CPU/mem/GPU consumption
slin nodemap [-p PART] [-S STATE]	Compact availability map + state summary
slin cpus [-p PART]	CPU usage and availability per partition
slin mem [-p PART] [-S STATE]	Memory usage per node
slin gpu [-p PART]	GPU nodes: allocated versus configured
slin fair	Fairshare and priority (sshare)
slin details JOBID	One job in detail, including per-step usage
slin watch [-v VIEW] [-i SECONDS] [-u USER] [-p PART]	Live-refreshing view

slin jobs -u alice -S PENDING            # alice's pending jobs
slin jobs -p gpu,bigmem -n 20            # first 20 jobs on two partitions
slin nodes --gpu -S idle                 # idle GPU nodes
slin nodes --no-detail                   # skip scontrol; faster on large clusters
slin details 4815162_7                   # array task
slin details 4815162+0                   # heterogeneous job component
slin watch -v nodes -i 5                 # views: summary, jobs, nodes, mem, cpus

watch redraws in place with rich.live — it never clears the screen, so your scrollback survives.

Controlling jobs

Every mutating command confirms first unless you pass -y/--yes, and reports a result per job so a partial failure in a batch is visible.

Command	Effect
slin cancel JOBID... [-s SIGNAL] [-y]	Cancel jobs, or send them a signal
slin hold JOBID... [-y]	Hold pending jobs
slin release JOBID... [-y]	Release held jobs
slin requeue JOBID... [-y]	Requeue jobs

slin cancel 4815162
slin cancel 4815162 4815163 4815164 --yes
slin cancel 4815162 --signal USR1        # signal instead of cancelling
slin hold 4815170 && slin release 4815170

Exit status is non-zero if any job in the batch failed, so && chains behave.

History and statistics

The database keeps job history past the cluster's accounting retention, plus per-step resource usage (MaxRSS, AveCPU) that sacct only exposes if you ask precisely.

Command	Purpose
slin sync [--since W] [--full] [--no-snapshot]	Import accounting into the local DB
slin snapshot	Record one point-in-time sample of jobs, nodes and partitions
slin history [FILTERS]	Read stored history (paged)
slin stats [--since W] [--by user|partition|both] [-n N]	Aggregates
slin prune [--keep W] [--keep-jobs W] [--vacuum]	Apply retention
slin daemon [-i SECONDS] [--sacct-every S] [--once] [-v]	Background poller
slin db info | path | vacuum	Inspect and maintain the database

slin sync --since 90d              # first backfill
slin sync                          # afterwards: resumes from the stored watermark
slin sync --full --since 30d       # re-import the window from scratch

slin history --me --since 7d
slin history -S FAILED --since 30d --limit 100
slin history --search train_bert --page 2
slin history --refresh --me        # sync first, then read
slin history --live --since 3d     # bypass the DB, query sacct directly

slin stats --since 30d --by partition
slin db info                       # row counts, coverage, sync watermarks
sqlite3 "$(slin db path)" 'select count(*) from job_samples'

Every write is an idempotent upsert, and the sync watermark only advances after the transaction commits — re-running slin sync, or interrupting it, is always safe.

history filters: -u/--user, -p/--partition, -S/--state, --since, --search, --me, -n/--limit, --page, --refresh, --live.

The daemon

slin works with nothing resident — sync and snapshot are on-demand. The daemon is opt-in, and exists to build the resource time series that per-node history and the TUI sparklines are drawn from.

slin daemon                                   # snapshot every 60s, sacct every 900s
slin daemon -i 30 --sacct-every 600 --verbose
slin daemon --once                            # one cycle and exit — for cron

It holds a PID file in $XDG_STATE_HOME/slin, shuts down cleanly on SIGTERM, and logs to $XDG_STATE_HOME/slin/daemon.log. For cron or a systemd --user timer, use --once:

*/5 * * * * slin daemon --once

TUI

slin tui                    # opens on the Jobs tab
slin tui --tab nodes        # jobs | nodes | partitions | history
slin tui --interval 10      # auto-refresh every 10s
slin tui --demo             # a synthetic cluster, no SLURM required

Key	Action
1 2 3 4	Jobs / Nodes / Partitions / History tab
r	Refresh now
p	Pause auto-refresh
/	Focus the filter bar (esc leaves it)
d	Toggle theme
q	Quit
Jobs	enter details · c cancel · h hold · e release · s cycle sort
Nodes	g GPU nodes only · b busy nodes only
History	]/[ page · w cycle time window · m only my jobs

Textual renders fine over SSH, so run it on the login node.

Demo mode

slin tui --demo runs the interface against a fake cluster generated by slin.demo: 26 nodes across four partitions, a live queue that keeps moving (jobs finish, pending jobs start, new ones arrive), a month of accounting history and twelve hours of per-node samples for the sparklines. Handy on a laptop, for screenshots, and for trying the key bindings before pointing slin at a real cluster.

It never runs a SLURM command: c/h/e only move the fake queue, and the history goes to slin-demo.db next to your real database, which is left untouched.

---

Scripting

--json turns every command into a data source; the payload is the same model layer the tables are built from.

slin --json jobs -S PENDING | jq -r '.[] | [.job_key, .username, .reason] | @tsv'
slin --json nodes --gpu     | jq -r '.[] | "\(.name) \(.gres_used) of \(.gres_total)"'
slin --json stats --since 30d | jq '.users[0]'

# cancel everything of yours that is stuck pending on a dead reason
slin --json myjobs -S PENDING \
  | jq -r '.[] | select(.reason == "PartitionDown") | .job_key' \
  | xargs -r slin cancel --yes

Exit codes: 0 success, 1 "nothing found" or a refused confirmation, 2 a SLURM or environment failure, 130 interrupted.

---

Configuration

$XDG_CONFIG_HOME/slin/config.toml (i.e. ~/.config/slin/config.toml). Both a flat table and a [slin] section are accepted.

[slin]
db_path = "~/.local/share/slin/slin.db"
scope = "auto"              # auto | all | user
command_timeout = 30        # seconds per SLURM call
refresh_interval = 5        # seconds, used by `watch` and the TUI
sample_retention_days = 90  # pruning of the *_samples tables
job_retention_days = 0      # 0 = keep job records forever
theme = "default"

Precedence, highest first: CLI options → SLIN_* environment variables → config file → built-in defaults.

export SLIN_DB_PATH=/scratch/$USER/slin.db
export SLIN_SCOPE=user
export SLIN_COMMAND_TIMEOUT=60
slin --config ./cluster-b.toml summary        # or point at a different file entirely

Paths follow the XDG spec: database in $XDG_DATA_HOME/slin, config in $XDG_CONFIG_HOME/slin, PID and log in $XDG_STATE_HOME/slin.

Scope

Not every cluster lets you see everyone's jobs. With scope = auto (the default) slin asks cluster-wide first and, if the cluster refuses, transparently retries as -u $USER and labels the view "your jobs only" — it never silently under-reports. Force the behaviour with --scope all or --scope user.

---

Development

python -m venv .venv && source .venv/bin/activate
pip install -e '.[dev]'

pytest                 # parser, db, render and TUI tests
ruff check slin tests
ruff format slin tests

The parsers are pure str -> model functions, so they are tested against real output recorded once on a cluster rather than a simulated SLURM. scripts/record_fixtures.sh captures it into tests/fixtures/ (usernames pseudonymised by default; set SLIN_ANONYMISE=0 to keep them):

./scripts/record_fixtures.sh              # run this on a login node

Keep the awkward cases represented when you replace a fixture — array and heterogeneous jobs, Unknown timestamps, (null) GRES, per-node/per-CPU ReqMem suffixes, scontrol values containing spaces — or the regression coverage goes with them.

The architecture, database schema and roadmap are documented in SCHEME.md. The short version: parsers return dataclasses, and the renderers, the database writer and the TUI all consume those same models — that shared data path is what keeps the three frontends consistent.

License

MIT — see LICENSE.