slurmwatch

Live, per-process CPU / memory / GPU telemetry for a running Slurm job — with a plain-language efficiency verdict.

CI PyPI Python 3.10+ MIT License Ruff

slurmwatch live TUI dashboard: per-process CPU, memory, and GPU telemetry for a Slurm job. Memory climbs from safe into the OOM-guard WARNING and CRITICAL bands while the allocation-efficiency verdict flags an idle GPU (1 of 2 active).

Features

  • Efficiency verdict — grades CPU, memory, and GPU (GOOD / UNDERUSED / IDLE / WARNING) so you know when you're wasting cores or GPUs.
  • Per-process GPU attribution — NVML sees only your PIDs, so a neighbor's job never inflates your numbers.
  • Honest memory — working set (RSS minus reclaimable cache) with a configurable OOM guard.
  • Works anywhere — full live telemetry on the node; auto-falls back to Slurm accounting (sstat) from a login node.
  • Zero config — slurmwatch <jobid> auto-discovers jobs, cgroup v1/v2, and whether it's on the node. No flags to memorize.

Install

pip install slurmwatch

Requires Python 3.10+ and Linux with cgroup v1 or v2. One install works across a mixed cluster: GPU monitoring (NVIDIA, via pynvml) auto-activates on GPU nodes and is silently skipped on CPU-only nodes. Works with pipx / uv too — e.g. uv tool install slurmwatch.

Usage

slurmwatch                       # auto-discover and attach to your running job
slurmwatch 12345                 # attach to a job (array: 12345_3, het: 12345+1)
slurmwatch --demo                # try the live TUI right now — no Slurm needed
slurmwatch 12345 --once --json   # one machine-readable snapshot, then exit
slurmwatch 12345 --log run.jsonl # headless logging (JSON Lines or CSV)

Run it from anywhere: on a login node, slurmwatch automatically attaches to the job's compute node (via srun --overlap) to show the live dashboard — no manual srun needed. If it can't attach, it falls back to an sstat summary (peak memory + CPU time + allocation); GPU utilization isn't available that way, since Slurm tracks GPU count, not per-device util. Set SLURMWATCH_NO_HOP=1 to skip the hop and always get the summary. (The attached view runs inside the job's allocation, so it counts against the job's resources.)

TUI keys: c/m/g/v focus a panel, arrows/PgUp/PgDn scroll, q quits.

Exit codes: 0 success · 1 runtime failure · 2 bad config. Errors go to stderr, so piped --once/--log output stays clean.

See slurmwatch --help for the full flag list. Behavior is also tunable via SLURMWATCH_* environment variables — e.g. SLURMWATCH_OOM_WARN, SLURMWATCH_GPU_IDLE_PCT, SLURMWATCH_POLL_INTERVAL (plus ASCII mode and more).

Library

import asyncio
from slurmwatch import TelemetryCollector, resolve_job_context

async def sample(job_id: str):
    collector = TelemetryCollector(resolve_job_context(job_id))
    await collector.start()
    try:
        print((await collector.next_snapshot()).to_json())
    finally:
        await collector.stop()

asyncio.run(sample("12345"))

Limitations

  • NVIDIA-only GPU support (no AMD/ROCm).
  • Single-node view — multi-node jobs show data for the node you're on.
  • Live GPU utilization and working-set memory require running on the job's node.

License

MIT

Project links

Data verified by PyPI on Jul 4, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (1.1.1)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for PursuitOfDataScience from gravatar.com PursuitOfDataScience

Credits

Author: Youzhi Yu

GitHub Statistics

Data verified by PyPI on Jul 4, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License expression

MIT
View SPDX License List

Requires

Python >=3.10

Provides Extra

nvidia dev

Tags

slurm hpc monitoring gpu tui

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

slurmwatch-0.1.2.tar.gz (2.5 MB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

slurmwatch-0.1.2-py3-none-any.whl (38.1 kB view details)

Uploaded Python 3

File details

Details for the file slurmwatch-0.1.2.tar.gz.

File metadata

  • Download URL: slurmwatch-0.1.2.tar.gz
  • Upload date:
  • Size: 2.5 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for slurmwatch-0.1.2.tar.gz
Algorithm Hash digest
SHA256 b513082055e57c99ba53facaa5c1dc5e38099166d69847b7b344303e69f391aa
MD5 21997054c6e0f2fad1d43d63f3be5568
BLAKE2b-256 ccf982446cb8b5bfbbc414c00db844986c9e763586c65d1269a1509dd023461e

See more details on using hashes here.

Provenance

The following attestation bundles were made for slurmwatch-0.1.2.tar.gz:

Publisher: release.yml on PursuitOfDataScience/slurmwatch

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file slurmwatch-0.1.2-py3-none-any.whl.

File metadata

  • Download URL: slurmwatch-0.1.2-py3-none-any.whl
  • Upload date:
  • Size: 38.1 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for slurmwatch-0.1.2-py3-none-any.whl
Algorithm Hash digest
SHA256 50922c7e717ec61707191c57f4719c3c459b705a71ab92514a6d2f61eac55c51
MD5 9c735c11b074ec027e5e8a310e951d11
BLAKE2b-256 f8142b77712b5407ccfd8e1e4118b0e8083bc82fc345a46eea814e04b6ef5cf6

See more details on using hashes here.

Provenance

The following attestation bundles were made for slurmwatch-0.1.2-py3-none-any.whl:

Publisher: release.yml on PursuitOfDataScience/slurmwatch

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

1.1.1

2 files

1.1.0

2 files

1.0.1

2 files

1.0.0

2 files

0.10.1

2 files

0.10.0

2 files

0.9.9

2 files

0.9.8

2 files

0.9.7

2 files

0.9.6

2 files

0.9.5

2 files

0.9.4

2 files

0.9.3

2 files

0.9.2

2 files

0.9.1

2 files

0.9.0

2 files

0.8.2

2 files

0.8.1

2 files

0.8.0

2 files

0.7.0

2 files

0.6.0

2 files

0.5.0

2 files

0.4.0

2 files

0.3.0

2 files

0.2.3

2 files

0.2.2

2 files

0.2.1

2 files

0.2.0

2 files

This release

0.1.2 This release

2 files

0.1.1

2 files

0.1.0

2 files

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page