slurmwatch

Live per-process CPU / memory / GPU telemetry for a running Slurm job — the facts, so you can judge.

CI PyPI Python 3.10+ MIT License Ruff

slurmwatch live TUI dashboard showing per-process CPU, memory, and GPU telemetry for a Slurm job.

Install

pip install slurmwatch      # or: uv tool install slurmwatch / pipx install slurmwatch

Python 3.10+ on Linux (cgroup v1/v2). GPU monitoring (pynvml) auto-activates on NVIDIA nodes and is skipped on CPU-only ones.

Usage

slurmwatch                       # auto-discover and attach to your running job
slurmwatch 12345                 # a specific job (array 12345_3, het 12345+1)
sw 12345                         # "sw" is a short alias
slurmwatch --demo                # live TUI, no Slurm needed
slurmwatch --demo pending        # preview the pending-job view offline
slurmwatch 12345 --once --json   # one machine-readable snapshot, then exit
slurmwatch 12345 --log run.jsonl # headless logging (JSONL or CSV)

Point it at a pending job and, instead of an error, you get why it's waiting, when it should start, and where it could run — no flags needed.

Keys — c/m/g open a full-screen CPU/memory/GPU drill-in (in the GPU view ↑/↓ pick a device) · type a node number (or ←/→) to switch node · p reveal a truncated path · ↑/↓ PgUp/PgDn scroll · q back/quit.

Notes

  • From a login node it attaches to the compute node via srun --overlap, so the view runs inside your allocation.
  • Can't attach? You get an sstat summary — peak memory, CPU time, allocation — but no live GPU utilization, which Slurm doesn't track per device.
  • SLURMWATCH_NO_HOP=1 forces the summary · --ascii for a non-UTF-8 terminal · SLURMWATCH_MOUSE=1 enables the wheel (off by default so text selection works).
  • Everything else: slurmwatch --help and the SLURMWATCH_* env vars.

Features

  • Facts, not verdicts — labelled bars (usage · used · compute · vram), each with its recent 60-second range and a health dot (●/▲/✖). An alarm strip surfaces only what needs action (MEMORY 91% of limit, 1 OF 2 GPUS IDLE).
  • Drill in — c/m/g open a focused full-screen view of one resource: the live figure in large digits plus a 60-second area chart of its recent history. In the GPU view, arrow through devices and the chart follows the one you pick.
  • Per-process — NVML and cgroups count only your PIDs, so a neighbour on a shared node never inflates your numbers.
  • Honest memory — working set (RSS minus reclaimable cache), against a configurable OOM guard.
  • Multi-node — one process, every node: type a node's number (or step with ◂ ▸) to switch which node the dashboard shows — jump straight to node 199 of a 200-node job.
  • Pending jobs — point it at a queued job and instead of an error you get why it's waiting (the Slurm reason, in plain English), when the scheduler estimates it'll start, and where it could run right now — a cluster-wide view of free capacity that flags any partition your request would fit into, with the exact scontrol update to requeue there.
  • Runs anywhere — full live telemetry on the node; falls back to Slurm accounting (sstat) when it can't attach.
  • Zero config — auto-discovers the job, cgroup v1/v2, GPUs, and where it's running.

Project links

Data verified by PyPI on Jul 12, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (1.1.1)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for PursuitOfDataScience from gravatar.com PursuitOfDataScience

Credits

Author: Youzhi Yu

GitHub Statistics

Data verified by PyPI on Jul 12, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License expression

MIT
View SPDX License List

Requires

Python >=3.10

Provides Extra

nvidia dev

Tags

slurm hpc monitoring gpu tui

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

slurmwatch-0.7.0.tar.gz (3.8 MB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

slurmwatch-0.7.0-py3-none-any.whl (101.9 kB view details)

Uploaded Python 3

File details

Details for the file slurmwatch-0.7.0.tar.gz.

File metadata

  • Download URL: slurmwatch-0.7.0.tar.gz
  • Upload date:
  • Size: 3.8 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for slurmwatch-0.7.0.tar.gz
Algorithm Hash digest
SHA256 813a4b69c21fec5cf2b6f1a2a59b1e0fd8eef535b82ee3865ced6ec9dbc2832d
MD5 b477695d2adb314650ec5260585a4041
BLAKE2b-256 cabbef39f466dd036b0cde209540967d7e2215feabcef6ce384900d1103bfb52

See more details on using hashes here.

Provenance

The following attestation bundles were made for slurmwatch-0.7.0.tar.gz:

Publisher: release.yml on PursuitOfDataScience/slurmwatch

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file slurmwatch-0.7.0-py3-none-any.whl.

File metadata

  • Download URL: slurmwatch-0.7.0-py3-none-any.whl
  • Upload date:
  • Size: 101.9 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for slurmwatch-0.7.0-py3-none-any.whl
Algorithm Hash digest
SHA256 36fad25d100d9344e16b62d041adf7b200b64162bf5e0688342993238169061a
MD5 739e77edc004a6cae13522630fd98469
BLAKE2b-256 7ec9913c52f175f572746e5ad435a3bace443ab966f94c36d1ddc2304d3a28d1

See more details on using hashes here.

Provenance

The following attestation bundles were made for slurmwatch-0.7.0-py3-none-any.whl:

Publisher: release.yml on PursuitOfDataScience/slurmwatch

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

1.1.1

2 files

1.1.0

2 files

1.0.1

2 files

1.0.0

2 files

0.10.1

2 files

0.10.0

2 files

0.9.9

2 files

0.9.8

2 files

0.9.7

2 files

0.9.6

2 files

0.9.5

2 files

0.9.4

2 files

0.9.3

2 files

0.9.2

2 files

0.9.1

2 files

0.9.0

2 files

0.8.2

2 files

0.8.1

2 files

0.8.0

2 files

This release

0.7.0 This release

2 files

0.6.0

2 files

0.5.0

2 files

0.4.0

2 files

0.3.0

2 files

0.2.3

2 files

0.2.2

2 files

0.2.1

2 files

0.2.0

2 files

0.1.2

2 files

0.1.1

2 files

0.1.0

2 files

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page