gnvitop

Global nvitop — a web-based GPU & TPU monitoring dashboard that monitors all your remote accelerator servers from a single page.

Like nvitop, but for all your servers at once — NVIDIA GPUs, MetaX GPUs, Google Cloud TPUs, and Gadi NCI compute nodes, displayed as a beautiful web dashboard.

pip install gnvitop
gnvitop

How It Works

1. Monitors local GPU/TPU automatically (no config needed)
2. Reads your ~/.ssh/config and SSH into each remote server
3. Auto-detects accelerator type: runs nvidia-smi (NVIDIA), mx-smi (MetaX), or checks /dev/accel* (Google TPU)
4. Displays everything in a real-time web dashboard with per-user process highlighting
5. Auto-refreshes every 30 seconds; SSE streaming shows each server as it responds

graph LR
    A[gnvitop] --> B[Browser]
    B --> C["localhost — Local GPUs"]
    B --> D["lab-server — 4x A100"]
    B --> E["metax-server — MetaX C500"]
    B --> F["tpu-v4-8 — Google TPU v4"]
    B --> G["gadi — NCI HPC (dynamic nodes)"]
    B --> H["offline-server — error"]

    style A fill:#7c3aed,stroke:none,color:#fff,font-weight:bold
    style B fill:#2563eb,stroke:none,color:#fff
    style C fill:#16a34a,stroke:none,color:#fff
    style D fill:#16a34a,stroke:none,color:#fff
    style E fill:#16a34a,stroke:none,color:#fff
    style F fill:#7c3aed,stroke:none,color:#fff
    style G fill:#16a34a,stroke:none,color:#fff
    style H fill:#dc2626,stroke:none,color:#fff

Installation

pip install gnvitop

Usage

gnvitop                              # start and auto-open browser
gnvitop -p 8080                      # custom port
gnvitop --host 0.0.0.0               # expose to LAN
gnvitop --no-browser                 # don't auto-open browser
gnvitop --ssh-config /path/to/config # custom SSH config
gnvitop --tui                        # terminal UI mode (no browser)
gnvitop --tui --tui-refresh 10       # TUI with 10s refresh interval
gnvitop --agent                      # output JSON for scripting/agents
gnvitop --history --csv out.csv      # record GPU history to CSV
gnvitop -v                           # show version

Or run as a module:

python -m gnvitop

Prerequisites

1. SSH config — your ~/.ssh/config should have server entries:

Host gpu-server-01
    HostName 192.168.1.101
    User alice
    IdentityFile ~/.ssh/id_rsa

Host gpu-server-02
    HostName 192.168.1.102
    User bob

# ProxyJump (bastion/jump host) is fully supported
Host compute-node
    HostName compute-node.internal
    User alice
    ProxyJump bastion-host

# Google Cloud TPU VM
Host tpu-v4-8
    HostName <external-ip>
    User <your-user>
    IdentityFile ~/.ssh/google_compute_engine

2. SSH key auth — password-less login should be set up
3. Accelerator tools — nvidia-smi (NVIDIA), mx-smi (MetaX), or /dev/accel* (TPU) on the remote servers

Features

• Zero config — reads ~/.ssh/config automatically, no setup needed
• One command — pip install gnvitop && gnvitop, that's it
• Local + Remote — monitors local accelerator alongside all remote servers
• Multi-vendor — supports NVIDIA GPUs (nvidia-smi), MetaX GPUs (mx-smi), and Google Cloud TPUs
• Non-bash shell safe — wraps remote commands in bash -c so it works even if the remote login shell is fish, zsh, etc.
• TPU support — detects Google Cloud TPU chips via /dev/accel*, shows chip count and HBM spec (v4: 32 GB/chip); utilization shown as N/A until torch_xla is installed
• MetaX support — parses mx-smi output for MetaX C500 and compatible GPUs
• Gadi NCI support — SSHes into Gadi login nodes and auto-discovers allocated GPU compute nodes via qstat
• ProxyJump support — monitors compute nodes behind bastion/jump hosts
• Per-GPU users — shows which users occupy each GPU and their memory usage
• User highlight — your own processes are highlighted in blue for quick identification
• Agent mode — gnvitop --agent outputs structured JSON for use in scripts and AI agents
• History recording — gnvitop --history records GPU stats to CSV for trend analysis
• TUI mode — gnvitop --tui for a terminal UI without a browser
• Auto browser — opens dashboard in your browser on start
• Adjustable refresh — choose 5s / 10s / 30s / 5min auto-refresh interval
• Concurrent — queries all servers in parallel (20 workers)
• Fast loading — background cache warming so the dashboard loads instantly
• Collapse cards — fold individual server cards to a compact strip
• Drag to reorder — drag server cards to arrange them in any order, persisted across reloads
• Compact / Normal modes — toggle between full detail and compact views
• Dark UI — clean, responsive dark-themed dashboard
• At a glance — summary bar shows online hosts, total GPUs, idle GPUs, free memory
• Color coded — green (online), purple (TPU), yellow (no GPU), red (offline), blue (local)

Agent Mode

gnvitop --agent outputs a JSON array suitable for scripting or AI agent use:

gnvitop --agent

[
  {
    "host": "gpu-server-01",
    "status": "ok",
    "gpus": [
      {
        "index": 0,
        "name": "NVIDIA A100-SXM4-80GB",
        "memory_total_mb": 81920,
        "memory_used_mb": 1200,
        "memory_free_mb": 80720,
        "gpu_utilization_pct": 3.0,
        "available": true
      }
    ]
  },
  {
    "host": "tpu-v4-8",
    "status": "ok",
    "gpus": [
      {
        "index": 0,
        "name": "Google TPU v4",
        "memory_total_mb": 32768,
        "memory_used_mb": -1,
        "memory_free_mb": -1,
        "gpu_utilization_pct": -1,
        "available": true
      }
    ]
  }
]

For TPU chips, memory_used_mb and gpu_utilization_pct are -1 (unknown) until torch_xla is installed on the TPU VM. available is true when no Python processes are detected.

Comparison with nvitop

gnvitop is not a replacement for nvitop — it's a complement. Use nvitop for detailed local process-level GPU monitoring, use gnvitop to get an overview of all your accelerator servers (including local) from one place.

License

MIT