xpustat

Authors: Salah Eddine Bekhouche, Hichem Telli

A unified accelerator stat tool: gpustat, but for every vendor.

Supported backends

Legend: ✅ tested on real hardware; 🔲 implemented, awaiting a hardware test

Backends whose management tool is not installed are silently skipped.

Apple Silicon note: the GPU shares unified memory with the CPU, so mem_total_mb is the machine's total RAM and mem_used_mb is the slice currently held by the GPU driver. macOS exposes GPU temperature, power, and per-process usage only to root (powermetrics), so those columns show N/A on Macs. Discrete GPUs in Intel Macs report real VRAM.

Neuron / Tenstorrent note: neuron-ls and tt-smi report inventory, processes and telemetry but not live memory usage, so those backends show memory as 0 / N/A.

Running Vendor Hardware Tests

If you have access to specific hardware, you can help us verify the implementation by running the standalone test scripts found in the vendor_devices_tests/ directory:

# Example: Testing the Huawei backend on a machine with Ascend NPUs
python vendor_devices_tests/test_huawei.py

# Example: Testing the NVIDIA backend
python vendor_devices_tests/test_nvidia.py

If the test successfully outputs your hardware metrics, please let us know so we can update the backend status to ✅ Tested!

Documentation

The full documentation is in docs/: installation, command line, Python API, device selection, Prometheus, backends and contributing. mkdocs serve renders it as a site.

Install

pip

pip install xpustat

conda

conda install -c conda-forge xpustat

CLI usage

xpustat                      # coloured table of all detected devices
xpustat -pui                 # watch mode, showing processes (-p), users (-u), refreshing every 1.0s (-i)
xpustat -i 2                 # watch mode, refresh every 2 seconds
xpustat -i 1 -n 10           # 10 refreshes then exit
xpustat -a                   # show everything (util, temp, power, arch, user, pid, cmd)
xpustat -f nvidia amd        # show only NVIDIA and AMD
xpustat --no-color           # plain text, no ANSI codes
xpustat --json               # raw JSON output
xpustat --prometheus         # Prometheus text format, once
xpustat --serve-prometheus 9400   # serve http://0.0.0.0:9400/metrics (fresh query per scrape)
xpustat --version

Example output:

xpustat 0.1.0  ·  myhost  ·  Fri May 22  11:24:00  2026
──────────────────────────────────────────────────────────────────────────
[NVIDIA] #0    Tesla V100                          │   1,024 / 32,768 MB   ███░░░░░░░░░░░░░    3%
[AMD   ] #0    Radeon RX 7900 XTX                  │      26 / 24,560 MB   ░░░░░░░░░░░░░░░░    0%
[APPLE ] #0    Apple M5                            │     375 / 32,768 MB   ░░░░░░░░░░░░░░░░    1%
[HUAWEI] #0:0  Ascend 310                          │   2,703 /  8,192 MB   █████░░░░░░░░░░░   33%

Example output with processes, users, and watch mode (xpustat -pui):

xpustat 0.1.0  ·  salah-amd  ·  Fri May 29  10:07:57  2026
──────────────────────────────────────────────────────────
[AMD   ] #0    Radeon RX 7900 XTX              │  17,918 / 24,560 MB  ████████████░░░░   72%
              └─ salah/348504(17480M)
  refreshing every 1.0s  ·  Ctrl+C to quit

Python API

import xpustat

# Query everything at once (parallel by default)
stats = xpustat.query_all()

print(stats)                      # XPUStatCollection(nvidia=0, amd=1, ...)
print(stats.amd[0].name)          # "Radeon RX 7900 XTX"
print(stats.amd[0].mem_used_mb)   # 26

# As a dict (JSON-serialisable)
import json
print(json.dumps(stats.to_dict(), indent=2))

# Query a single vendor
from xpustat import NvidiaGPUStatCollection
gpus = NvidiaGPUStatCollection.new_query()

Picking a device for training / inference

Every device object carries vendor-neutral fields on top of the vendor ones:

import xpustat

dev = xpustat.pick_device()[0]            # most free memory first, honours *_VISIBLE_DEVICES
model.to(dev.torch_device)                # "cuda:2", "npu:0", "mps", ...

xpustat.pick_device(n=2, by="util")       # two least-busy devices
xpustat.pick_device(vendor="amd", min_free_mb=8000)
xpustat.visible_indices("nvidia")         # parsed CUDA_VISIBLE_DEVICES, or None when unset

Prometheus / Grafana

xpustat --serve-prometheus 9400           # then scrape http://host:9400/metrics

Exposes xpustat_memory_{used,total,free}_bytes, xpustat_utilization_percent, xpustat_temperature_celsius, xpustat_power_watts, xpustat_process_memory_bytes and xpustat_device_info (with arch and torch_device labels) for every vendor from one process. xpustat.to_prometheus(stats) returns the same text for embedding in your own exporter.

Version history

0.3.0

• Apple Silicon GPU support (apple backend) read in-process from IOKit (~2 ms, ioreg fallback), no root required; tested on Apple M5
• -A shows the GPU core count for Apple devices (e.g. 10-core)
• Fix: vendor collections now support index access (stats.amd[0]), as documented
• Faster subprocess fallbacks: the nvidia-smi and amd-smi invocations now run concurrently
• Watch mode no longer swallows errors silently
• Fix: process lines now render as user:command/pid instead of running user and command together
• The nvidia / fast extras now pull nvidia-ml-py (the official NVML bindings) instead of the deprecated pynvml package
• New backends: AWS Neuron (neuron-ls), Google Cloud TPU (tpu-info), Tenstorrent (tt-smi)
• Vendor-neutral device fields: vendor, device_index, mem_free_mb, framework_index, torch_device
• xpustat.pick_device() picks the least-loaded device(s), honouring *_VISIBLE_DEVICES
• Prometheus export: --prometheus, --serve-prometheus PORT, and xpustat.to_prometheus()

0.2.0

• Fix AMD GPUs not being detected when amd-smi reports "N/A" sections or idle GPUs (e.g. ROCm 10 with an iGPU present)
• Flicker-free watch mode: frames are redrawn in place on the alternate screen

0.1.0

• Initial release
• Support for NVIDIA, AMD, Intel Gaudi, Intel Arc/GPU, Huawei Ascend, Hygon DCU, Cambricon MLU, Moore Threads