nvidb

A package that provides an aggregated view of the NVIDIA GPU information on several hosts.

1. Installation

1.1 Install using pip

You can install nvidb using pip. First, clone the repository:

git clone https://github.com/FanBB2333/nvidb.git
cd nvidb
pip install .

Or install directly from PyPI:

pip install nvidb
# If the specified version is unavailable in your custom repository, use pypi.org as the source:
pip install nvidb -i https://pypi.org/simple

1.2 Configuration

Option A: Interactive Setup (Recommended)

Use the interactive command to add servers:

nvidb add

This will guide you through adding a new server with prompts for host, port, username, authentication method, etc.

Option B: Manual Configuration

To manually configure remote servers, create or edit the configuration file at ~/.nvidb/config.yml:

mkdir -p ~/.nvidb/
cp config.example.yml ~/.nvidb/config.yml
# Edit the file with your server details

Configuration file template:

servers:
  - hostname: "example1.com"       # Server hostname or IP address
    port: 22                       # SSH port number
    username: "user1"              # SSH username for authentication
    nickname: "Production GPU"     # Human-readable nickname for display
    auth: "auto"                   # Authentication method: auto | key | password
    identityfile: "~/.ssh/id_ed25519"  # Optional, used only when auth is auto/key

Configuration Options:

• hostname: Server hostname or IP address (required)
• port: SSH port, default is 22 (required)
• username: SSH username (required)
• nickname: Human-readable server nickname (optional)
• auth: Authentication method - auto, key, or password (optional, default: auto)
• identityfile: SSH private key path (optional, only effective when auth is auto or key)
• password: SSH password (optional, will prompt if needed)

Warning: Storing passwords in plaintext in the configuration file is NOT RECOMMENDED for security reasons. Consider using SSH key-based authentication (auth: key) instead.

Environment Variables

You can customize the working directory by setting NVIDB_HOME:

export NVIDB_HOME=/path/to/custom/nvidb

Default working directory is ~/.nvidb/.

2. Usage

2.1 Basic Commands

nvidb                  # Monitor local GPU only (interactive TUI)
nvidb --remote         # Monitor local and remote servers
nvidb --once           # Print GPU stats once and exit
nvidb --once --remote  # Print all servers once and exit
nvidb --version        # Show version

Tip: Set remote: true under the basic section of ~/.nvidb/config.yml to make plain nvidb include remote servers by default (same for nvidb log). Pass --no-remote for a one-off local-only run.

GPU status is collected directly from NVML. Local collection uses nvidia-ml-py; remote collection keeps a standard-library Python agent open over SSH and calls libnvidia-ml.so.1 directly, so no Python package needs to be installed on the remote host. nvidia-smi -q -x is retained only as a compatibility fallback when NVML cannot be initialized.

2.2 Server Management

nvidb add              # Interactively add a new server
nvidb import [path]    # Import servers from SSH config (default: ~/.ssh/config)
nvidb info             # Show configuration info and server list

2.3 GPU Logging

Continuously log GPU statistics to an SQLite database:

nvidb log                          # Log local GPU with default settings
nvidb log --remote                 # Log local and remote GPUs
nvidb log --interval 10            # Set logging interval to 10 seconds
nvidb log --db-path /path/to/db    # Specify custom database path

Press Ctrl+C to stop logging and save data.

2.4 Web Dashboard

Open a Dash-based interactive web dashboard to view live GPU info and browse log sessions:

pip install dash
nvidb web                 # Web dashboard (Live + Logs)
nvidb web --db-path /path/to/db
nvidb web --port 8502

After the server starts (http://localhost:8501 by default):

• Live: per-server GPU tables plus rolling utilization / VRAM charts; toggle include remote, pick the refresh interval, or pause auto-refresh. (basic.remote: true or nvidb --remote web enables remote by default.)
• Logs: pick a session in the left table, then filter by node / metric / time range. Charts support zoom, pan and legend isolation; click any chart point to inspect that snapshot. The raw table supports filtering, sorting and CSV export.

nvidb log web is deprecated; use nvidb web instead.

2.5 Cleanup

Remove server configurations or delete log data:

nvidb clean              # Interactive cleanup menu
nvidb clean all          # Delete all data (requires double confirmation)

2.6 Interactive TUI Navigation

When viewing GPU stats, use these keyboard shortcuts:

2.7 GPU Monitor Decorator

Use the @nvidb.monitor decorator to track GPU usage during function execution:

import nvidb

@nvidb.monitor
def train_model():
    # Your training code here
    pass

# With custom options
@nvidb.monitor(sample_interval=0.05, gpu_indices=[0, 1])
def multi_gpu_training(epochs: int = 100):
    pass

# Async function support
@nvidb.monitor
async def async_training():
    pass

After function execution, it outputs:

======================================================================
[nvidb.monitor] Function completed: train_model
  Signature: train_model()
  Location: /path/to/file.py:14
----------------------------------------------------------------------
  Duration: 125.3s
----------------------------------------------------------------------
  GPU 0: NVIDIA GeForce RTX 3090 Ti
    Memory:
      Peak:    8192.00 MiB / 24.00 GiB
      Delta:   +6144.00 MiB
    Utilization:
      Avg:     85.0%
    Temperature:
      Peak:    72C
    Power:
      Peak:    320.5W
======================================================================

Decorator Options:

• sample_interval: Sampling interval in seconds (default: 0.1)
• gpu_indices: List of GPU indices to monitor (default: all GPUs)
• enabled: Enable/disable monitoring (default: True)

4. System Requirements

• NVIDIA driver with NVML (libnvidia-ml.so.1)
• Python 3.8+
• Python 3.8+ and SSH access on remote servers
• nvidia-smi is optional and used only as an NVML failure fallback

5. Tips

• The live header shows Source: nvml during normal collection and Source: nvidia-smi if the compatibility fallback was needed
• Database files are stored in ~/.nvidb/gpu_log.db by default
• Configuration and logs are stored in ~/.nvidb/ directory

6. Show me the screenshots

• Monitor local info with nvidb:

• Monitor remote info with nvidb --remote:

• Monitor on web panel with nvidb web:

Local info:

Remote info:

7. Acknowledgements

• Thanks to NVIDIA for providing NVML and nvidia-ml-py, used for direct GPU telemetry collection.
• Thanks to nvitop for demonstrating efficient direct NVML polling and metric caching patterns.
• Thanks to NVIDIA for providing nvidia-smi, retained as a compatibility fallback.
• Thanks to Paramiko for powering SSH connections for remote monitoring.
• Thanks to PyYAML for YAML-based configuration loading and saving.
• Thanks to pandas for parsing and processing GPU stats and log data.
• Thanks to blessed for building the interactive terminal UI.
• Thanks to termcolor for colored terminal output.
• Thanks to Streamlit for providing the web dashboard framework.