xputop

An interactive, universal accelerator and system monitor for terminal environments, supporting Huawei Ascend NPUs, NVIDIA GPUs, AMD GPUs, Intel XPUs, Google TPUs, and Custom backends. Inspired by nvitop and nvtop.

Features

• Universal Multi-Backend Support: Seamlessly monitors Huawei NPU, NVIDIA, AMD, Intel, and Google TPU. Supports configuring fallback or custom backends right from the configuration file using regex extractors !
• Rich Terminal UI: Beautiful per-device cards, memory usage bars, and active process lists.
• Fixed process row count per card to prevent UI jitter (-p N, default 3)
• Sparkline history curves (like nvtop) for Hardware, CPU, memory, disk, and network metrics
• Detail mode (-d) for combined chart + CPU + memory + disk view
• Full detail mode (-a) adds network I/O monitoring and system process tree
• Unified Hardware Telemetry: Automatically displays Power (W) and AICore/GPU Core Utilization (%) alongside VRAM constraints consistently across NPU, AMD, Intel, and NVIDIA.
• Background Async Logging: Run xputop -o logs.jsonl to silently record all metrics into JSONL/CSV datasets in the background without blocking or affecting training throughput!
• Offline HTML Visualization: Render your captured log datasets as a beautiful interactive HTML dashboard natively using xputop view logs.jsonl.
• Auto-Kill / Memory Safety Engine: Automatically SIGKILL top memory-consuming processes when VRAM or System RAM exceeds critical thresholds (Supports top N processes, dropping to an explicit limit, or targeted killing).
• System Telemetry: CPU per-core utilization, System memory (RAM + Swap), Disk usage paths, Network I/O.
• Threshold-based email alerting for both accelerator and system metrics.
• Configuration stored securely via xputop config.
• Lightweight design — zero extra subprocesses, strict timeout handles, instantaneous CPU sampling, and zero blocking I/O buffering.

Installation

# Minimal version (Hardware probing only)
pip install xputop

# Full feature-set (Includes system-monitoring tools & generic Database writers)
pip install "xputop[all]"

# Install with generic SQL database support only
pip install "xputop[db]"

Quick Start

# Launch the interactive TUI
xputop

# Detail mode: inline sparklines + CPU + memory + disk
xputop -d

# Enable gorgeous full 2D NVTop-style matrix line plotting (defaults to GPU and Memory)
xputop -P 

# Merge all GPUs and CPU onto a single unified 2D Plot overlay!
xputop -M -P cpu,gpu,mem

# Switch to grayscale or light themes
xputop -P --theme grayscale
xputop -M -P --theme light

# Responsive Layouts (Normal 40-wide, Compact 20-wide, Grid Alignment)
xputop -w 120         # Override default 80-width to 120 (fits 3 regular cards per row)
xputop -x             # Enable extreme compact mode (Fits 4 dense cards within 80-char width)
xputop -x -g          # Force strict grid alignment instead of Masonry-style wrapping layout

# Record in CSV natively
xputop -d -o metrics.csv

# Record trace into SQLite database (Native, zero-dependency)
xputop -o metrics.sqlite --format sqlite

# Send logging data traces to remote MySQL DB! (Requires pip install 'xputop[db]')
xputop -o mysql+pymysql://user:pass@host/db

# Wait and let the format inference determine it natively based on suffix!
xputop -o logs.jsonl

# Automatically infer current date, saving to /logs/xputop_log_xxxxxxx.csv
xputop -o /logs/

# Control process display rows (default 3, 0=hide, -1=show all)
xputop -n 5 -P
xputop -n 0           # hide all processes

# Monitor specific disk paths
xputop -C -D /data /home

# Probe and list available backend states
xputop --list-backends
xputop --reset-backend

# Force specific backend (Will permanently cache setting to bypass auto-timeout logic on large clusters)
xputop --backend npu
xputop --backend nvidia

# Run in demo mode (no hardware needed)
xputop --demo -d

# Print a single snapshot and exit
xputop once --cpu --mem --disk

# Print snapshot as JSON
xputop once --json --cpu --mem --disk /data

# Show Chinese help
xputop --zh

# Generate a default configuration file
xputop config --generate

Command-Line Options

Custom Hardware Support

xputop supports writing flexible standard Regex patterns direct inside ~/.config/xputop/config.toml to extract monitoring variables from any CLI binary!

[backend.custom.my_chip]
probe_command = "custom-smi --version"
info_command = "custom-smi log --csv"
device_pattern = "Device (?P<id>\\d+) \\| (?P<name>\\S+) \\| Temp=(?P<temp>\\d+) \\| Power=(?P<power>[\\d.]+) \\| Util=(?P<util>\\d+)% \\| Mem=(?P<mem_used>\\d+)/(?P<mem_total>\\d+)"

Build & Release

See BUILD.md for development setup, building, and publishing instructions.

Requirements

• Python >= 3.8

License

Apache License 2.0