slurmwatch

Live per-process CPU / memory / GPU telemetry for a running Slurm job — the facts, so you can judge.

CI PyPI Python 3.10+ MIT License Ruff

slurmwatch live TUI dashboard showing per-process CPU, memory, and GPU telemetry for a Slurm job.

Install

pip install slurmwatch      # or: uv tool install slurmwatch / pipx install slurmwatch

Python 3.10+ on Linux (cgroup v1/v2). GPU monitoring (pynvml) auto-activates on NVIDIA nodes and is skipped on CPU-only ones.

Usage

slurmwatch                       # auto-discover and attach to your running job
slurmwatch 12345                 # a specific job (array 12345_3, het 12345+1)
sw 12345                         # "sw" is a short alias
slurmwatch --demo                # live TUI, no Slurm needed
slurmwatch 12345 --once --json   # one machine-readable snapshot, then exit
slurmwatch 12345 --log run.jsonl # headless logging (JSONL or CSV)
slurmwatch 12345                 # PENDING job? shows why it's waiting + where it could run

Keys — c/m/g CPU/memory/GPU detail · type a node number (or ◂ ▸) switch node · p reveal a truncated path · ↑ ↓ PgUp PgDn scroll · q quit.

Notes

  • From a login node it attaches to the compute node via srun --overlap, so the view runs inside your allocation.
  • Can't attach? You get an sstat summary — peak memory, CPU time, allocation — but no live GPU utilization, which Slurm doesn't track per device.
  • SLURMWATCH_NO_HOP=1 forces the summary · --ascii for a non-UTF-8 terminal · SLURMWATCH_MOUSE=1 enables the wheel (off by default so text selection works).
  • Everything else: slurmwatch --help and the SLURMWATCH_* env vars.

Features

  • Facts, not verdicts — labelled bars (usage · used · compute · vram), each with its recent 60-second range and a health dot (●/▲/✖). An alarm strip surfaces only what needs action (MEMORY 91% of limit, 1 OF 2 GPUS IDLE).
  • Per-process — NVML and cgroups count only your PIDs, so a neighbour on a shared node never inflates your numbers.
  • Honest memory — working set (RSS minus reclaimable cache), against a configurable OOM guard.
  • Multi-node — one process, every node: type a node's number (or step with ◂ ▸) to switch which node the dashboard shows — jump straight to node 199 of a 200-node job.
  • Pending jobs — point it at a queued job and instead of an error you get why it's waiting (the Slurm reason, in plain English), when the scheduler estimates it'll start, and where it could run right now — a cluster-wide view of free capacity that flags any partition your request would fit into, with the exact scontrol update to requeue there.
  • Runs anywhere — full live telemetry on the node; falls back to Slurm accounting (sstat) when it can't attach.
  • Zero config — auto-discovers the job, cgroup v1/v2, GPUs, and where it's running.

Project links

Data verified by PyPI on Jul 12, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (1.1.1)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for PursuitOfDataScience from gravatar.com PursuitOfDataScience

Credits

Author: Youzhi Yu

GitHub Statistics

Data verified by PyPI on Jul 12, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License expression

MIT
View SPDX License List

Requires

Python >=3.10

Provides Extra

nvidia dev

Tags

slurm hpc monitoring gpu tui

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

slurmwatch-0.5.0.tar.gz (3.8 MB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

slurmwatch-0.5.0-py3-none-any.whl (97.0 kB view details)

Uploaded Python 3

File details

Details for the file slurmwatch-0.5.0.tar.gz.

File metadata

  • Download URL: slurmwatch-0.5.0.tar.gz
  • Upload date:
  • Size: 3.8 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for slurmwatch-0.5.0.tar.gz
Algorithm Hash digest
SHA256 8aa4a2ac5c94a3b31a1da728fdcc4e5965dfb44ee661d42ee2a214991a8448da
MD5 0a349c5664aa6deff7d712dabaa8e4d7
BLAKE2b-256 d607b403a3e0dc85903a9c16e79d246b7e2cc42c78e8cd9d30bd06d9db25c64c

See more details on using hashes here.

Provenance

The following attestation bundles were made for slurmwatch-0.5.0.tar.gz:

Publisher: release.yml on PursuitOfDataScience/slurmwatch

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file slurmwatch-0.5.0-py3-none-any.whl.

File metadata

  • Download URL: slurmwatch-0.5.0-py3-none-any.whl
  • Upload date:
  • Size: 97.0 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for slurmwatch-0.5.0-py3-none-any.whl
Algorithm Hash digest
SHA256 92ea7b0332e5679129c95d8da98ac4cedc0b6e124b13d8cb14eaf376ea9b57d1
MD5 8e39b2b33e2c72b6cc4f4f379df99639
BLAKE2b-256 becb2566d160d9789ecda5b47b480d489b8d9faddbd763772fc665bf3a30f0f6

See more details on using hashes here.

Provenance

The following attestation bundles were made for slurmwatch-0.5.0-py3-none-any.whl:

Publisher: release.yml on PursuitOfDataScience/slurmwatch

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

1.1.1

2 files

1.1.0

2 files

1.0.1

2 files

1.0.0

2 files

0.10.1

2 files

0.10.0

2 files

0.9.9

2 files

0.9.8

2 files

0.9.7

2 files

0.9.6

2 files

0.9.5

2 files

0.9.4

2 files

0.9.3

2 files

0.9.2

2 files

0.9.1

2 files

0.9.0

2 files

0.8.2

2 files

0.8.1

2 files

0.8.0

2 files

0.7.0

2 files

0.6.0

2 files

This release

0.5.0 This release

2 files

0.4.0

2 files

0.3.0

2 files

0.2.3

2 files

0.2.2

2 files

0.2.1

2 files

0.2.0

2 files

0.1.2

2 files

0.1.1

2 files

0.1.0

2 files

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page