scopos 4.0.1

A Textual terminal UI that monitors NVIDIA GPUs grouped by user — live memory/utilisation, per-user breakdown, tmux process management, scopos-API progress bars, and a confirm-guarded batch kill.

A user-centric NVIDIA GPU monitor for the terminal

About

SCOPOS is a Textual-based terminal UI for monitoring NVIDIA GPUs, grouped by user so you can tell at a glance who is using what. Beyond live memory, utilisation and temperature, it offers focused Zen and Tmux views, a lightweight Python API for your training scripts to report live status (progress bars with an ETA, loss, stage, …), and a guarded kill mode for cleaning up stray processes — right-click to copy, batch-select to terminate. The layout and theme are tunable from a single config file.

• Python: 3.9+

Installation

Install with pipx

pipx installs the application in an isolated environment while making the command globally available.

pip install pipx
pipx ensurepath

pipx install --python python3.9 scopos

Quick Start

monitor all GPUs

scopos

highlight user "alice" and show their task details

scopos -u alice

refresh every 2 seconds

scopos -i 2

synthetic data, no NVIDIA driver needed

scopos --demo

start in zen (focus) mode

scopos -u alice --zen

---

Tabs / modes

A tab bar under the logo switches between four views (cycle with m, or jump with g / z / t / i; start on one with -m/--mode):

• Global — every GPU, every user (the classic layout).

• Zen — focused on -u/--user (see below).
• Tmux — your own tmux processes in one flat, grid-style table (same columns and interactions as the cards), grouped by session:win.pane. Idle pane shells are dimmed so the programs actually running stand out. Right-click a row to copy it; in danger mode you can also kill the process, its whole pane, or its whole session (multi-process kills list every affected process and ask for confirmation).

• Info — scopos version and this host's basic specs (CPU, RAM, GPUs).

tmux sockets are per-user, so the Tmux tab shows the tmux server of the user running scopos.

Zen mode

Switch to zen mode (z, or start with -m zen), a focused layout meant to be paired with -u/--user:

• Each GPU's table lists only the watched user's processes.
• The per-GPU bar and legend still show every user — the watched user is highlighted (★, bold) so you keep the full picture at a glance.
• The table drops the USER and S.START columns and instead shows the live fields each process reports through the Python API below — including animated progress bars.
• A resident CPU card lists every process of the watched user that reports to scopos but isn't currently on a GPU — extending the monitor to plain CPU jobs (e.g. data preprocessing) and to jobs still importing CUDA / loading data. It shows host RAM instead of GPU memory; once a job allocates GPU memory it simply appears under its GPU as well.

Every process row shows both MEM/GB (GPU memory) and RAM/GB (host memory). The SESSION column shows the tmux session name (tmux:<name>) for tmux-managed processes — for your own sessions; other users' tmux sockets aren't readable, so those fall back to tmux. Determinate progress bars also show an ETA (· ~3m 20s) estimated from how fast the bar is advancing.

Mouse & shortcuts

• Hover any cell to see its full, untruncated content as a tooltip. Built-in columns (like COMMAND) are width-capped to keep rows compact; user-reported metadata columns are always shown in full.
• Right-click a process row for a menu: Copy row info copies that row's fields to the clipboard.
• Danger mode (ctrl+shift+k) is an independent toggle that works in every mode. While it is on, the right-click menu also offers Kill, which shows the full details and asks for confirmation before sending a terminate signal. The status bar shows a red ⚠ DANGER reminder while it is armed.
• Batch kill: in danger mode a checkbox column appears; click it to tick rows (ticked rows float to the top so they stay visible across refreshes, and the row cursor is preserved too). Right-click → Kill N selected to kill them together. Press c to clear all ticks.

The bottom status bar also shows scopos's own CPU% / memory footprint.

Tuning the layout & theme

All the cosmetic knobs live in one place — src/scopos/config.py. You can either edit that file, or override any of it without touching the source by dropping a config.toml (or config.json) into ~/.scopos (honours $SCOPOS_HOME). Only the keys you list are overridden. Restart scopos after changing anything.

Layout / spacing

• COLUMN_WIDTHS — per-column width caps (clipped cells show …; None = auto-size). Metadata columns are always shown in full.
• COLUMN_VISIBLE — show/hide any built-in column.
• TABLE_CELL_PADDING — the gap between columns.
• CARD_MIN_WIDTH / CARD_MAX_WIDTH — how wide GPU cards get (and thus how many tile per row).
• GRID_GUTTER / GRID_PADDING / CARD_PADDING — spacing around and inside cards.
• TABLE_MAX_HEIGHT — how tall a table grows before it scrolls.

Colours / theme

• USER_PALETTE / WATCH_USER_COLOR — per-user colours and the watched user's colour.
• PROGRESS_COLOR / BAR_TRACK_COLOR — progress-bar fill and track.
• COLOR_OK / COLOR_WARN / COLOR_CRIT and the *_WARN / *_CRIT thresholds — the green/yellow/red status colours for GPU free memory, the host RAM meter and temperature.

Example ~/.scopos/config.toml:

card_min_width = 90
table_cell_padding = 2

[column_widths]
COMMAND = 30

[column_visible]
"S.START" = false

[colors]
progress = "magenta"
watch_user = "bright_blue"

(TOML needs Python 3.11+, or the tomli package on older versions; config.json always works.)

Python API

scopos doubles as a tiny library so your scripts can push live status to the monitor. Importing it is cheap — no Textual or NVIDIA driver required.

import scopos

# Report plain fields (merged into this process's metadata):
scopos.report(stage="train", loss=0.1234, acc="92.5%")

# Report a progress bar. scopos renders it as a live bar in zen mode:
for step in range(total_steps):
    scopos.report(progress=scopos.progress(step, total_steps))  # e.g. 37/100
    ...

# A fraction in [0, 1] works too, and an indeterminate (animated) bar:
scopos.report(loading=scopos.progress())            # bouncing "…"
scopos.report(warmup=scopos.progress(0.5, label="halfway"))

# Drop a field by reporting None; replace everything with set(...):
scopos.report(loss=None)
scopos.set(stage="done")

# Or scope a run and clean up automatically:
with scopos.session(stage="train"):
    train()   # metadata file removed on exit

Each process writes ~/.scopos/metadata/<pid>.json; scopos reads it back and, in zen mode, shows every reported field as a column next to that process. The file is removed automatically when your program exits (atexit). Set $SCOPOS_HOME to relocate the .scopos directory.

---

Requirements

• Python >= 3.9
• textual >= 8.0
• psutil >= 7.0
• nvidia-ml-py >= 12.0

License

See LICENSE in the repository.

Links

• Homepage/Repo
• Issues