nvmon

A fancy NVIDIA GPU monitor for the terminal. Single Python file, no dependencies.

It fits the window. In a small one the boxes get shorter, and scroll once they no longer fit:

p lists every job; a picked one stands out on the GPUs it uses:

v goes from graphs to each GPU's processes in their place, one a line with the command, to the numbers alone, to GPU and MEM alone. The wheel scrolls the processes, sideways too (tilted, with Shift, or ← →):

Install

uv tool install nvmon    # or: pipx install nvmon

No uv? Install it, or use pip install nvmon inside whichever environment you use.

On a server without internet, copy nvmon.py over and run it with any Python 3.6+.

Usage

nvmon              # Esc, q or Ctrl+C to quit
nvmon -i 1         # refresh every second (default: 0.5)
nvmon -g 0,2,4-7   # only these GPUs
ssh -t HOST nvmon  # over ssh: -t gives it a terminal

Click a process, or press Tab, to pick its job: the GPUs it uses stand out, and f shows only those. t asks the job to stop (SIGTERM: it may save and exit), k ends it at once (SIGKILL); both ask for a y first. p lists every job with its utilization and command line; click a heading to sort by it. Click an empty spot to let the job go and close the list.

The GPU numbers at the bottom left hide a GPU when clicked, and show it again; so does the number's key. v goes through graphs, processes, numbers only and GPU and MEM only (V back). c goes through the themes (C back): nvmon's own, then palettes from works by Hiroshige, Hokusai, O'Keeffe, Van Gogh, Bénédictus and Cassatt (as MetBrewer has them) and the Rosé Pine theme's, then black and white. The theme, graphs on or off, the list's order and the GPUs hidden stay for the next run, for each computer you connect from (in ~/.config/nvmon/settings.json). The keys work with a Korean keyboard in Hangul mode too (ㅂ quits). Hold Shift to select text with the mouse.

What it shows

• GPU: how much of the time something ran on it. cores / tensor (H100 and newer): how much of the chip that work kept busy, and how much of it was matrix math on the Tensor Cores. A GPU reads 100 % as soon as one small kernel runs all the time, while its cores may sit at 30 %.
• MEM, data sent CPU -> GPU and GPU -> CPU against the link's top speed, power, temperature, fan and clock.
• Processes on each GPU with run time and memory, grouped by account and conda environment (your own account unlabelled), script names for Python.
• Warnings only when something is off: SLOWED: power cap | too hot | hw brake when the clock is held back, PCIe DEGRADED: x16 -> x8 when the card runs on fewer lanes than it supports.

Notes

• Needs an NVIDIA driver.
• On Windows, run it in Windows Terminal; Git Bash's default window is not a terminal to Python.
• nvmon asks the terminal how it draws. One that puts line and block characters two cells wide (Xshell, PuTTY and the like, set up for Korean, Chinese or Japanese) gets boxes and graphs drawn in ASCII. One that does not say it shows 24-bit color gets 256 colors, which look nearly the same.
• Boxes still broken: the terminal moves on one cell, but its font draws the characters two wide. Pick a font whose line characters are one cell wide, and make sure the encoding is UTF-8.
• When a newer release is out, the bottom line says so, with the command that updates your install. nvmon asks PyPI once at start; NVMON_NO_UPDATE_CHECK=1 turns that off.

License

MIT