jobgpumonitor-server

The consumer side of jobgpumonitor: reads the JSONL events written by the emitter and the scheduler probe, keeps one document per run, sends notifications (ntfy, Telegram, webhook) and serves a small read-only API.

jobgpumonitor (in the job) ──┐
                             ├──> $JGM_DIR/runs/**.jsonl ──> jgmd serve ──> ntfy / Telegram / webhook
jgm scheduler (login node) ──┘                                   └──> SQLite ──> HTTP API

Quick start (login node)

pip install "jobgpumonitor-server[api]"
jgmd init                         # writes ~/.config/jgm-server/config.toml
$EDITOR ~/.config/jgm-server/config.toml   # put an ntfy topic or a Telegram bot in [notify.*]
jgmd notify-test                  # phone should buzz
jgmd serve --api                  # keep it in tmux / systemd --user

Then, still on the login node, jgm scheduler from the emitter package so that OOM, time-outs, preemptions and the queue are reported too.

What you get notified about

Every alert fires once per run and is recorded, so restarting the server never re-sends. A channel that fails (ntfy down, no network) gets the alert again later, with backoff, for up to 6 hours; the other channels are not sent it twice. Raw events are kept keep_events_days after they are received (not after they were emitted).

API

jgmd serve --api (needs the [api] extra) exposes on 127.0.0.1:21834:

GET /health
GET /runs?phase=running
GET /runs/<cluster>/<job>/<restart>
GET /runs/<cluster>/<job>/<restart>/events?after=0&types=metric.log,progress.update
GET /runs/<cluster>/<job>/<restart>/stream          # server-sent events
GET /runs/<cluster>/<job>/<restart>/logs?stream=stdout   # the job's .out/.err, live
GET /alerts

Interactive docs at /docs. From your laptop: ssh -L 21834:localhost:21834 cluster.

CLI

jgmd serve [--once] [--api]    ingest, rules, notifications
jgmd runs [--phase running]    table of runs
jgmd show <run_id | job id>    full run document (--events N for raw events)
jgmd alerts                    what was sent, and through which channel
jgmd notify-test               send a test message
jgmd init                      write the example config

Configuration lives in ~/.config/jgm-server/config.toml (see jgmd init); JGMD_DIRS, JGMD_NTFY_TOPIC, JGMD_TELEGRAM_TOKEN / JGMD_TELEGRAM_CHAT_ID, JGMD_WEBHOOK_URL work without a file.

Development

uv venv && uv pip install -e ".[dev]" && uv run pytest