Skip to main content

No project description provided

Project description

nvidia-smi2

A tool for enriching the output of nvidia-smi.

Alt text

Usage

Install

pip install nvidia-smi2

Run

nvidia-smi2 [-l [length]] [-u [username]]
  print GPU utilization with usernames and CPU stats for each GPU-utilizing process

  -l|--command-length [length]     Print longer part of the commandline. If `length'
                                   is provided, use it as the commandline length,
                                   otherwise print first 100 characters.
  -c|--color                       Colorize the output (green - free GPU, yellow -
                                   moderately used GPU, red - fully used GPU)
  -u|--user                        Name of user to summarize

Or run from src

pip install termcolor
chmod a+x nvidia-htop.py
./nvidia-htop.py [-l [length]] [-u [username]]
  print GPU utilization with usernames and CPU stats for each GPU-utilizing process

  -l|--command-length [length]     Print longer part of the commandline. If `length'
                                   is provided, use it as the commandline length,
                                   otherwise print first 100 characters.
  -c|--color                       Colorize the output (green - free GPU, yellow -
                                   moderately used GPU, red - fully used GPU)
  -u|--user                        Name of user to summarize

Note: for backward compatibility, nvidia-smi | ./nvidia-htop.py [-l [length]] [-u [username]] is also supported.

Note: running inside a container (docker, singularity, ...), nvidia-smi can only see processes running in the container.

Example output

rnd@rnd:~$ nvidia-smi2 -l
Wed Jul 12 10:41:16 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  On   | 00000000:01:00.0 Off |                  N/A |
| 55%   73C    P2    88W / 280W |   3248MiB / 11176MiB |     64%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce ...  On   | 00000000:02:00.0 Off |                  N/A |
| 59%   67C    P2    91W / 280W |   9397MiB / 11178MiB |      7%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
|  GPU   PID     USER    GPU MEM  %CPU  %MEM      TIME  COMMAND                                                                                               |
|    0  1242     root       9MiB   0.0   0.0  175 days  /usr/lib/xorg/Xorg vt1 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx  |
|    0  1966      gdm       4MiB   0.0   0.0  175 days  /usr/bin/gnome-shell                                                                                  |
|    0  6963  anonym1    2365MiB   2.9   3.1    7 days  python train.py                                                                                       |
|    0  8200     root     135MiB   0.1   0.4  117 days  python3 app.py xxxxxxxxx                                                                              |
|    0 12474      rnd     135MiB   0.1   0.8   11 days  python3 main.py xxxxxxxxxxx                                                                           |
|    0 19369     root     455MiB  62.4   2.5   20 days  python3 read_rtsp.py --path_stream xxxxxxxxxxxxxxxxxx                                                 |
|    0 30502     root     135MiB   0.0   1.8  16:24:10  python build_index.py                                                                                 |
|    1  1242     root       4MiB   0.0   0.0  175 days  /usr/lib/xorg/Xorg vt1 -displayfd 3 -auth xxxxxxx                                                     |
|    1 19422     root     455MiB  86.7   3.7   20 days  python3 read_rtsp.py --path_stream xxxxxxxxxxxxxxxxxxxxxxx                                            |
|    1 25255  anonym2    6917MiB   128   4.3  22:30:29  python train_image_classifier.py xxxx                                                                 |
|    1 27877     root    2017MiB   9.4   1.4   15 days  python consumer.py                                                                                    |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
|      USER    TOTAL GPU MEM             TOTAL %CPU  TOTAL %MEM  |
+----------------------------------------------------------------+
|      root            13MiB                    0.0         0.0  |
|       gdm             4MiB                    0.0         0.1  |
|    anony1          1086MiB                    0.5         3.2  |
|    anony2          6930MiB                  478.8         9.5  |
|    anony3          7418MiB                  111.0         5.4  |
|----------------------------------------------------------------|
| all-users         15451MiB / 22528MiB       590.3        18.2  |
+----------------------------------------------------------------+
rnd@rnd:~$ nvidia-smi2 -l -u root
Wed Jul 12 10:51:06 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.57.02    Driver Version: 470.57.02    CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  On   | 00000000:01:00.0 Off |                  N/A |
| 58%   69C    P2   106W / 280W |   3248MiB / 11176MiB |      5%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce ...  On   | 00000000:02:00.0 Off |                  N/A |
| 59%   65C    P2    80W / 280W |   9397MiB / 11178MiB |      7%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
|  GPU   PID     USER    GPU MEM  %CPU  %MEM      TIME  COMMAND                                                                                               |
|    0  1242     root       9MiB   0.0   0.0  175 days  /usr/lib/xorg/Xorg vt1 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx  |
|    0  8200     root     135MiB   0.1   0.4  117 days  python3 app.py xxxxxxxxx                                                                              |
|    0 19369     root     455MiB  62.4   2.5   20 days  python3 read_rtsp.py --path_stream xxxxxxxxxxxxxxxxxx                                                 |
|    0 30502     root     135MiB   0.0   1.8  16:33:59  python build_index.py                                                                                 |
|    1  1242     root       4MiB   0.0   0.0  175 days  /usr/lib/xorg/Xorg vt1 -displayfd 3 -auth xxxxxxx                                                     |
|    1 19422     root     455MiB  86.7   3.7   20 days  python3 read_rtsp.py --path_stream xxxxxxxxxxxxxxxxxxxxxxx                                            |
|    1 27877     root    2017MiB   9.4   2.2   15 days  python consumer.py                                                                                    |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------+
|      USER    TOTAL GPU MEM             TOTAL %CPU  TOTAL %MEM  |
+----------------------------------------------------------------+
|      root          3210MiB                  158.6        10.6  |
|----------------------------------------------------------------|
| all-users         15451MiB / 22528MiB       590.3        18.2  |
+----------------------------------------------------------------+

Reference

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

nvidia_smi2-0.0.2.post6.tar.gz (5.7 kB view hashes)

Uploaded Source

Built Distribution

nvidia_smi2-0.0.2.post6-py3-none-any.whl (6.3 kB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page