nvitop 0.2.3

An interactive NVIDIA-GPU process viewer.

nvitop

An interactive NVIDIA-GPU process viewer.

This project is inspired by nvidia-htop, a tool for enriching the output of nvidia-smi. nvidia-htop uses regular expressions to read the output of nvidia-smi from a subprocess, which is inefficient. In the meanwhile, there is a powerful interactive GPU monitoring tool called nvtop. But nvtop is written in C, which makes it lack of portability. And What is really inconvenient is that you should compile it yourself during installation. Therefore, I made this repo. I got a lot help when reading the source code of ranger, the console file manager. Some files in this repo are copied and modified from ranger under the GPLv3 License.

This project is currently in the beta phase, most features have been tested on Linux. If you are using Windows with CUDA-capable GPUs, please submit feedback on the issue page, thank you very much!

If this repo is useful to you, please star ⭐️ it to let more people know 🤗.

Features

• Informative and fancy output: show more information than nvidia-smi with colorized fancy box drawing.
• Monitor mode: can run as a resource monitor, rather than print the results only once. (vs. nvidia-htop, limited support with command watch -c)
• Interactive: responsive for user inputs in monitor mode. (vs. py3nvml)
• Efficient:
  • query device status using NVML Python bindings directly and cache them with ttl_cache from cachetools, instead of parsing the output of nvidia-smi. (vs. nvidia-htop)
  • display information using the curses library rather than print with ANSI escape codes. (vs. py3nvml)
• Portable: work on both Linux and Windows.
  • get host process information using the cross-platform library psutil instead of calling ps -p <pid> in a subprocess. (vs. nvidia-htop & py3nvml)
  • written in pure Python, easy to install with pip. (vs. nvtop)

Requirements

• Python 3.5+
• NVIDIA Management Library (NVML)
• nvidia-ml-py
• psutil
• cachetools
• curses
• termcolor

Note: The NVIDIA Management Library (NVML) is a C-based programmatic interface for monitoring and managing various states. The runtime version of NVML library ships with the NVIDIA display driver (available at Download Drivers | NVIDIA), or can be downloaded as part of the NVIDIA CUDA Toolkit (available at CUDA Toolkit | NVIDIA Developer). The lists of OS platforms and NVIDIA-GPUs supported by the NVML library can be found in the NVML API Reference.

Installation

Install from PyPI:

$ pip install --upgrade nvitop

Install the latest version from GitHub (recommended):

$ pip install git+https://github.com/XuehaiPan/nvitop.git#egg=nvitop

Or, clone this repo and install manually:

$ git clone --depth=1 https://github.com/XuehaiPan/nvitop.git
$ cd nvitop
$ pip install .

Usage

Query the device and process status. The output is similar to nvidia-smi, but has been enriched and colorized.

# Query status of all devices
$ nvitop

# Specify query devices
$ nvitop -o 0 1  # only show <GPU 0> and <GPU 1>

# Only show devices in `CUDA_VISIBLE_DEVICES`
$ nvitop -ov

Run as a resource monitor, like htop:

# Automatically configure the display mode according to the terminal size
$ nvitop -m

# Arbitrarily display as `full` mode
$ nvitop -m full

# Arbitrarily display as `compact` mode
$ nvitop -m compact

# Specify query devices
$ nvitop -m -o 0 1  # only show <GPU 0> and <GPU 1>

# Only show devices in `CUDA_VISIBLE_DEVICES`
$ nvitop -m -ov

Press q to return to the terminal.

Type nvitop --help for more information:

usage: nvitop [-h] [-m [{auto,full,compact}]] [-o idx [idx ...]] [-ov]
              [--gpu-util-thresh th1 th2] [--mem-util-thresh th1 th2]

A interactive NVIDIA-GPU process viewer.

optional arguments:
  -h, --help            show this help message and exit
  -m [{auto,full,compact}], --monitor [{auto,full,compact}]
                        Run as a resource monitor. Continuously report query data,
                        rather than the default of just once.
                        If no argument is given, the default mode `auto` is used.
  -o idx [idx ...], --only idx [idx ...]
                        Only show the specified devices, suppress option `-ov`.
  -ov, --only-visible   Only show devices in environment variable `CUDA_VISIBLE_DEVICES`.
  --gpu-util-thresh th1 th2
                        Thresholds of GPU utilization to distinguish load intensity.
                        Coloring rules: light < th1 % <= moderate < th2 % <= heavy.
                        ( 1 <= th1 < th2 <= 99, defaults: 10 75 )
  --mem-util-thresh th1 th2
                        Thresholds of GPU memory utilization to distinguish load intensity.
                        Coloring rules: light < th1 % <= moderate < th2 % <= heavy.
                        ( 1 <= th1 < th2 <= 99, defaults: 10 80 )

Keybindings for monitor mode

• q: quit and return to the terminal.
• (<Left> <Right>) / ([ ]) / <S-Wheel>: scroll the host information of processes.
• <Home> / <C-a>: scroll the host information of processes to the beginning of line.
• <End> / <C-e>: scroll the host information of selected processes to the end of line.
• (<Up> <Down>) / (<Tab> <S-Tab>) / <Wheel>: select and highlight process.
• <Esc>: clear selection.
• T: send signal.SIGTERM to selected process.
• K: send signal.SIGKILL to selected process.
• <C-c>: send signal.SIGINT to selected process.

Note: Press CTRL key to multiply the mouse wheel events by 5.

Screenshots

Example output of nvitop:

Example output of nvitop -m:

Full	Compact

License

GNU General Public License, version 3 (GPLv3)