slurmwatch

Live per-process CPU / memory / GPU telemetry for a running Slurm job — the facts, so you can judge.

CI PyPI Python 3.10+ MIT License Ruff

slurmwatch live TUI dashboard showing per-process CPU, memory, and GPU telemetry for a Slurm job.

Install

pip install slurmwatch      # or: uv tool install slurmwatch / pipx install slurmwatch

Python 3.10+ on Linux (cgroup v1/v2). GPU monitoring (pynvml) auto-activates on NVIDIA nodes and is skipped on CPU-only ones.

Usage

slurmwatch                       # auto-discover and attach to your running job
slurmwatch 12345                 # a specific job (array 12345_3, het 12345+1)
sw 12345                         # "sw" is a short alias
slurmwatch --demo                # live TUI, no Slurm needed
slurmwatch --demo pending        # preview the pending-job view offline
slurmwatch 12345 --once --json   # one machine-readable snapshot, then exit
slurmwatch 12345 --log run.jsonl # headless logging (JSONL or CSV)

Point it at a pending job and, instead of an error, you get why it's waiting, when it should start, and where it could run — no flags needed.

Keys — c/m/g open a full-screen CPU/memory/GPU drill-in (in the GPU view ↑/↓ pick a device) · type a node number (or ←/→) to switch node · p reveal a truncated path · ↑/↓ PgUp/PgDn scroll · q back/quit.

Notes

  • From a login node it attaches to the compute node via srun --overlap, so the view runs inside your allocation.
  • Can't attach? You get an sstat summary — peak memory, CPU time, allocation — but no live GPU utilization, which Slurm doesn't track per device.
  • SLURMWATCH_NO_HOP=1 forces the summary · --ascii for a non-UTF-8 terminal · SLURMWATCH_MOUSE=1 enables the wheel (off by default so text selection works).
  • Everything else: slurmwatch --help and the SLURMWATCH_* env vars.

Features

  • Facts, not verdicts — labelled bars (usage · used · compute · vram), each with its recent 60-second range and a health dot (●/▲/✖). An alarm strip surfaces only what needs action (MEMORY 91% of limit, 1 OF 2 GPUS IDLE).
  • Drill in — c/m/g open a focused full-screen view of one resource: the live figure in large digits plus a 60-second area chart of its recent history. In the GPU view, arrow through devices and the chart follows the one you pick.
  • Per-process — NVML and cgroups count only your PIDs, so a neighbour on a shared node never inflates your numbers.
  • Honest memory — working set (RSS minus reclaimable cache), against a configurable OOM guard.
  • Multi-node — one process, every node: type a node's number (or step with ◂ ▸) to switch which node the dashboard shows — jump straight to node 199 of a 200-node job.
  • Pending jobs — point it at a queued job and instead of an error you get why it's waiting (the Slurm reason, in plain English), when the scheduler estimates it'll start, and where it could run right now — a cluster-wide view of free capacity that flags any partition your request would fit into, with the exact scontrol update to requeue there.
  • Runs anywhere — full live telemetry on the node; falls back to Slurm accounting (sstat) when it can't attach.
  • Zero config — auto-discovers the job, cgroup v1/v2, GPUs, and where it's running.

Project links

Data verified by PyPI on Jul 12, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (1.1.1)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for PursuitOfDataScience from gravatar.com PursuitOfDataScience

Credits

Author: Youzhi Yu

GitHub Statistics

Data verified by PyPI on Jul 12, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License expression

MIT
View SPDX License List

Requires

Python >=3.10

Provides Extra

nvidia dev

Tags

slurm hpc monitoring gpu tui

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

slurmwatch-0.6.0.tar.gz (3.8 MB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

slurmwatch-0.6.0-py3-none-any.whl (100.3 kB view details)

Uploaded Python 3

File details

Details for the file slurmwatch-0.6.0.tar.gz.

File metadata

  • Download URL: slurmwatch-0.6.0.tar.gz
  • Upload date:
  • Size: 3.8 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for slurmwatch-0.6.0.tar.gz
Algorithm Hash digest
SHA256 96b928b60cdff164734656d17315481420d2304a949b29693127f8bec6615ea9
MD5 15e89fcced7f329350b8cf568d4edcc8
BLAKE2b-256 dc9fc79c9efbb67c206b5311b91a3a5424b89d3f628b06178970840de5c946db

See more details on using hashes here.

Provenance

The following attestation bundles were made for slurmwatch-0.6.0.tar.gz:

Publisher: release.yml on PursuitOfDataScience/slurmwatch

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file slurmwatch-0.6.0-py3-none-any.whl.

File metadata

  • Download URL: slurmwatch-0.6.0-py3-none-any.whl
  • Upload date:
  • Size: 100.3 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for slurmwatch-0.6.0-py3-none-any.whl
Algorithm Hash digest
SHA256 e423666fd20bf8ac8f1f6680ef935e7fa10087be710aeb025c67303c1fe29619
MD5 f57f36a035692bfff9840233483be201
BLAKE2b-256 be0da188e6d3680a9b8e200c4740c676d22d541443a3e167a402b4d699286023

See more details on using hashes here.

Provenance

The following attestation bundles were made for slurmwatch-0.6.0-py3-none-any.whl:

Publisher: release.yml on PursuitOfDataScience/slurmwatch

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

1.1.1

2 files

1.1.0

2 files

1.0.1

2 files

1.0.0

2 files

0.10.1

2 files

0.10.0

2 files

0.9.9

2 files

0.9.8

2 files

0.9.7

2 files

0.9.6

2 files

0.9.5

2 files

0.9.4

2 files

0.9.3

2 files

0.9.2

2 files

0.9.1

2 files

0.9.0

2 files

0.8.2

2 files

0.8.1

2 files

0.8.0

2 files

0.7.0

2 files

This release

0.6.0 This release

2 files

0.5.0

2 files

0.4.0

2 files

0.3.0

2 files

0.2.3

2 files

0.2.2

2 files

0.2.1

2 files

0.2.0

2 files

0.1.2

2 files

0.1.1

2 files

0.1.0

2 files

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page