metax-mxtop

metax-mxtop is an nvitop-like terminal monitor for MetaX GPUs. Install the PyPI package, then run the mxtop command.

Built upon onlylrs/metax-mxtop, with thanks to the original project for the foundation.

Colors follow nvitop's intensity model: each device summary uses its dominant GPU or memory load, while the detailed MEM / MBW / UTL / PWR bars retain metric-specific colors (memory thresholds 10/80, GPU thresholds 10/75).

Idle — light load, mostly green	Mixed — per-bar color bands	Heavy — saturation across the cluster

Hosts with up to 16 GPUs retain the detailed nvitop-style list. Larger 32/64-GPU fleets use an adaptive overview grid in auto, compact, and one-shot views, while full mode keeps every detailed field.

Preview and gallery images use fixed-time deterministic synthetic MetaX-shaped telemetry so every state is reproducible. Live monitoring data still comes exclusively from the selected MetaX backend; optional fields in the fixtures illustrate presentation states rather than proving that every backend reports them.

Features

• nvitop-like terminal dashboard for MetaX GPUs, with full, compact, and automatic layouts.
• Read-only GPU telemetry through MXSML/Pymxsml or mx-smi.
• GPU device panel with temperature, power, utilization, memory, memory bandwidth, clocks, bus id, persistence, performance state, and driver fields when available.
• Adaptive 32/64-GPU fleet grid that preserves per-device load colors and leaves substantially more room for host/process data.
• Fleet memory totals in the title row — VRAM: used / total (pct%) across all visible GPUs and DRAM: used / total (pct%) for the host — colored by memory load and using no extra screen rows.
• Host panel with load average and scrolling history graphs for CPU, memory, swap, and the selected GPU's memory/utilization.
• Process table with selection, multi-process tagging, vertical and horizontal scrolling, and nvitop-compatible sorting.
• Process environment, host/GPU process-tree, per-process metrics, and built-in help screens.
• Confirmed SIGINT, SIGTERM, and SIGKILL actions for selected host processes, with --readonly to disable every process-changing action.
• ANSI-colored one-shot output, curses colors, ASCII fallback, and JSON output from the same filtered snapshot model.
• Shared device, user, PID, visibility, compute, and graphics filters across TUI, text, and JSON output.

GPU management remains read-only: mxtop does not update firmware, reset GPUs, or change persistence/admin settings. Interactive process signals are always presented for confirmation, validate process identity to guard against PID reuse, and enforce process ownership unless mxtop is running as root. Use --readonly when even confirmed host-process actions must be unavailable.

Install

From PyPI:

pip install -U metax-mxtop

The installed command is:

mxtop

From source:

git clone https://github.com/linkedlist771/metax-mxtop.git
cd metax-mxtop
pip install -e .

A man page ships with the package (man mxtop after a system-wide or user-scheme install); it is also readable in-repo with man -l docs/mxtop.1.

Shell completions are generated from the CLI itself:

mxtop --print-completion bash > ~/.local/share/bash-completion/completions/mxtop
mxtop --print-completion zsh  > ~/.zfunc/_mxtop     # ensure ~/.zfunc is in fpath
mxtop --print-completion fish > ~/.config/fish/completions/mxtop.fish

MetaX backend discovery

mxtop tries backends in this order when --backend auto is used:

1. Pymxsml/MXSML, when importable and usable.
2. mx-smi, when the CLI is available.

For the mx-smi backend, the executable path is resolved in this order:

1. Explicit backend path when constructed by callers.
2. MXTOP_MXSMI_PATH environment variable.
3. /opt/mxdriver/bin/mx-smi.
4. mx-smi from PATH.

Example:

MXTOP_MXSMI_PATH=/opt/mxdriver/bin/mx-smi mxtop --backend mxsmi

When GPUs don't show up, mxtop --doctor walks the whole discovery chain — Pymxsml importability and SDK wheels, mx-smi resolution, a live backend snapshot with device count, terminal capabilities, and config validity — printing a fix hint for anything that isn't a PASS.

Usage

On a terminal, the default command opens the interactive dashboard:

mxtop

Use --monitor / -m to request it explicitly or choose a layout:

mxtop -m
mxtop -m full
mxtop -m compact

An explicit monitor requires both stdin and stdout to be TTYs. When the default command is redirected or piped, it prints one snapshot instead. --once / -1 always selects one-shot text output:

mxtop --once

ANSI color is enabled for a terminal and can be disabled or forced:

mxtop --once --no-color
mxtop --once --force-color

--json prints one machine-readable snapshot without ANSI styling:

mxtop --json

--count / -n repeats one-shot or JSON output N times at the --interval cadence, for cron jobs and logging pipelines:

mxtop -n 10 --interval 5           # ten text snapshots, 5s apart
mxtop --json -n 60 --interval 1    # one minute of JSON samples
mxtop --json-lines -n 60 --interval 1 | jq '.devices[0].gpu_util_percent'

--json-lines / --ndjson emits one compact JSON object per line, the natural shape for jq, log shippers, and time-series ingestion.

For Prometheus-based monitoring of a single host, --export-metrics serves the local backend's telemetry on /metrics (default port 9532, dcgm-exporter style) — no SSH or remote extra needed. The exporter can terminate TLS directly:

mxtop --export-metrics --bind 0.0.0.0 --port 9532 --auth-token secret \
  --tls-cert /etc/mxtop/fullchain.pem --tls-key /etc/mxtop/privkey.pem
curl --cacert /etc/mxtop/ca.pem -H 'Authorization: Bearer secret' \
  https://monitor-host.example.com:9532/metrics

Common filters:

mxtop --only 0 1
mxtop --only-visible
mxtop --user alice bob
mxtop --user                 # current user
mxtop --pid 1234 5678
mxtop --compute
mxtop --only-compute
mxtop --graphics
mxtop --only-graphics

--compute and --graphics include processes with those contexts, including mixed X / C+G processes. Their uppercase --only-* forms require an exact compute-only or graphics-only type. Multiple supplied filters are combined. If a backend reports processes without context-type telemetry, as some mx-smi versions do, requesting one of these filters exits with a stable error instead of silently excluding every process.

Backend and layout options:

mxtop --backend auto
mxtop --backend pymxsml
mxtop --backend mxsmi
mxtop --interval 1.0
mxtop --no-unicode
mxtop --readonly

Useful CLI flags:

Environment variables

Configuration file

Persistent defaults live in ~/.config/mxtop/config.toml (or $XDG_CONFIG_HOME/mxtop/config.toml, or the file named by MXTOP_CONFIG). Explicit CLI flags and environment variables always take precedence over the file. Unknown or invalid keys print a warning instead of being silently ignored.

interval = 1.0            # refresh cadence in seconds (min 0.25)
monitor = "full"          # default layout: auto | full | compact
colorful = true           # spectrum bar colors
light = false             # light terminal theme
readonly = false          # disable process signals
no-unicode = false        # ASCII output only
gpu-util-thresh = [10, 75]
mem-util-thresh = [10, 80]

[remote]                  # --remote-mode defaults
bind = "127.0.0.1"
port = 8080
auth-token = "change-me"
tls-cert = "/etc/mxtop/fullchain.pem"
tls-key = "/etc/mxtop/privkey.pem"
# Optional, only for an encrypted private key:
tls-key-password-file = "/etc/mxtop/tls-key-password"
mxsmi-path = "mx-smi"
command-timeout = 10.0   # maximum seconds for each remote command
open = false

Direct HTTPS

The remote dashboard and local Prometheus exporter can terminate TLS directly. --tls-cert and --tls-key must always be supplied together; add --tls-key-password-file only when the private key is encrypted. The password file must contain exactly one non-empty line. The [remote] keys shown above provide the same settings for --remote-mode; pass the TLS flags explicitly to --export-metrics.

The certificate file must be a PEM full chain: the server certificate first, followed by any intermediate certificates. Its Subject Alternative Name (SAN) must cover every hostname or IP address clients use. The PEM private key must match that certificate. Restrict the key and optional password file to the mxtop service account (for example, mode 0600), and do not place their contents in the config file, command line, logs, or issue reports.

For a wildcard bind such as 0.0.0.0, mxtop prints and opens a localhost URL. Include localhost in the certificate SAN when using --open, or open the certificate's actual DNS name manually from remote clients.

Publicly trusted certificates work with normal browser and Prometheus trust stores. For a self-signed certificate, install that certificate as a trust anchor; for a private CA, install its CA certificate. Configure Prometheus with ca_file, and do not work around trust errors by disabling verification. Certificate, key, and password files are loaded once at startup, so restart mxtop after renewal or replacement.

TLS encrypts traffic but does not decide who may access telemetry. Keep using a high-entropy --auth-token, host firewall rules, and a narrowly scoped bind address. A TLS-terminating reverse proxy remains appropriate when it owns ACME renewal, certificate reload, mutual TLS, or organization-wide access policy. When a proxy terminates TLS, keep mxtop's HTTP listener on loopback and configure the proxy to append Secure to the upstream auth cookie; mxtop adds that flag itself only when it terminates TLS directly.

Remote mode (cluster dashboard)

Aggregate several SSH-reachable nodes into one local web dashboard:

pip install 'metax-mxtop[remote]'        # pulls in asyncssh
# Auto-discover concrete Host aliases from ~/.ssh/config which have mx-smi:
mxtop --remote-mode --open
# Add explicitly named nodes to the discovered set:
mxtop --remote-mode --discover --nodes nodeA nodeB --open
# Or monitor only an explicit set:
mxtop --remote-mode --nodes nodeA nodeB nodeC --open
# or list hosts in a file (one per line, # comments allowed)
mxtop --remote-mode --nodes-file ~/hosts.txt --port 8080
# Direct HTTPS with an operator-managed certificate:
mxtop --remote-mode --nodes nodeA nodeB --bind 0.0.0.0 --auth-token secret \
  --tls-cert /etc/mxtop/fullchain.pem --tls-key /etc/mxtop/privkey.pem --open

• With no --nodes or --nodes-file, mxtop enumerates concrete Host aliases in ~/.ssh/config (including Include files), probes them in parallel using key/agent authentication, and selects nodes where mx-smi -L reports at least one GPU. Wildcard Host patterns cannot be enumerated and are skipped. Use --discover to merge discovered nodes with an explicit list.

• Host names are plain SSH aliases resolved from your ~/.ssh/config (HostName/User/Port/IdentityFile/ProxyJump all honoured), so nodes with different MetaX card configurations work without extra setup.

• Authentication uses your SSH keys / ssh-agent — the "login once" model. For password-only nodes, run ssh-copy-id first; mxtop does not store passwords.

• The dashboard polls every node concurrently (--interval, default 2s), serves on 127.0.0.1 by default (--bind to change), and streams live updates over Server-Sent Events. Wildcard binds (0.0.0.0 / ::) print a usable localhost access URL separately from the all-interfaces listener; IPv6 access URLs are bracketed correctly. Every SSH telemetry command has a finite deadline (--remote-command-timeout, default 10s), so a hung command marks only that node down and reconnects it on the next sample instead of freezing updates for the whole fleet.

• Dashboard access can be protected with a shared token via --auth-token or the MXTOP_AUTH_TOKEN environment variable. Requests must then carry Authorization: Bearer <token>, or visit the printed HTTP or HTTPS URL with ?token=<token> once — the token is stored in a 24-hour HttpOnly cookie and is also marked Secure under direct HTTPS. When --bind exposes the dashboard beyond localhost without encryption or without a token, mxtop prints separate warnings: all cluster telemetry (hostnames, users, process command lines) would otherwise be exposed to the network or readable by anyone who can reach the port.

• A Prometheus endpoint at /metrics exports the latest cluster snapshot as text exposition: per-GPU utilization, memory, bandwidth, temperature, power, clocks, and ECC errors labelled by node/gpu/name/uuid, plus host CPU/RAM/load, per-node reachability (mxtop_node_up) and SSH collect latency. Point a Prometheus scrape_config at the dashboard (with the bearer token when set) for alerting and long-term history:

  scrape_configs:
    - job_name: mxtop
      static_configs:
        - targets: ["monitor-host:8080"]
      scheme: https
      authorization:
        type: Bearer
        credentials: <your --auth-token value>
      tls_config:
        # Omit ca_file for a certificate from a public CA.
        ca_file: /etc/prometheus/pki/mxtop-ca.pem
  

  The target name must be present in the server certificate's SAN. Keep certificate verification enabled; distribute the private CA certificate to Prometheus rather than setting insecure_skip_verify.

• The web UI provides a fleet overview with a switchable GPU heatmap, live trend sparklines (cluster GPU utilization, HBM, and host CPU; per-node GPU/HBM on the detail page), a searchable node inventory, cluster-wide host CPU/RAM/load telemetry, a process table, and per-node GPU, host, and process detail. Click a PID or command to open an investigation page with current values and rolling CPU, GPU-utilization, GPU-memory, and host-memory history. An exited process or unreachable node keeps its last sample visible; creation time or backend identity defines generations when available, with ended-state, command, and runtime-reset reconciliation covering remote PID reuse. A new generation never inherits the previous process's charts. Every node and process column is sortable by mouse or keyboard; the active direction remains visible and survives filters, navigation, and live updates, while unavailable values stay last. The header pause control or p/Z freezes the exact displayed snapshot across navigation, search, sorting, and JSON export; SSE ingestion and bounded history continue, buffered sample count remains visible, and resume applies the newest frame atomically. Bounded incident history also survives same-tab reloads for up to one hour when the dashboard runs in a secure browser context (HTTPS or browser-trusted loopback): it is stored as AES-GCM ciphertext partitioned by the monitored host set, while the key lives only in sessionStorage and is rotated on a new token bootstrap. On plain LAN HTTP, history remains in memory for the open page. Clear history removes the current tab session's retained trends and process records while preserving the current sample. Remote process rows are enriched with one batched ps query per node; missing Linux host fields degrade to unavailable values without taking the node offline. View state is encoded in the URL hash, so browser back/forward navigation works without reconnecting to the nodes. Dark and light themes follow the browser's prefers-color-scheme and can be switched with the header toggle; the choice persists in the browser.

• Each node runs the same mx-smi queries as the local backend; override the remote binary with --remote-mxsmi-path if it is not on PATH.

Interactive keys

General and navigation

Screens and process actions

Signals apply to all tagged processes when tags exist, otherwise to the selected process (or highlighted process-tree entry). The dialog supports Left/Right, Tab/Shift-Tab, ,/., mouse wheel, and button clicks; choose SIGTERM, SIGKILL, SIGINT, or Cancel, then confirm with Enter, Space, or y. Process actions require psutil and are disabled by --readonly.

Process sorting

Packaging and releases

The PyPI distribution name is metax-mxtop; the Python import package and CLI command are both mxtop.

pip install metax-mxtop
mxtop --version

Branch and pull-request runs execute the Python compatibility matrix and build and test the offline wheelhouse. Release publication is gated on the Python 3.9 wheelhouse checks, the full Python 3.10-3.13 test matrix, lint, and the Chromium dashboard suite. A push to main creates a commit-addressed prerelease; versioned GitHub and PyPI publication is limited to v* tags:

1. Bump pyproject.toml, src/mxtop/__init__.py, and the local package entry in uv.lock.
2. Regenerate the preview, gallery, and showcase assets and run their --check commands.
3. Run the full test, lint, and package-build commands below.
4. Commit the change.
5. Create a new semver tag, for example v0.1.22.
6. Push the commit and that tag.

GitHub Actions builds and Twine-validates the wheel and source distribution once, bundles them with the offline wheelhouse and a shared SHA256SUMS.txt, and makes both publishers download and verify that same workflow artifact. Automation refuses to overwrite an existing tagged GitHub Release. The minimal PyPI job performs no checkout or rebuild; it publishes the prebuilt distributions through OIDC Trusted Publishing.

More previews

See the output gallery and screen showcase for rendered stdout and deterministic secondary-screen fixtures across palettes, layouts, filters, edge telemetry, and 32/64-GPU fleets.

Security

See SECURITY.md for responsible disclosure and secure deployment guidance. Direct TLS or a TLS-terminating reverse proxy encrypts dashboard/exporter traffic; --auth-token, firewall policy, and a narrow bind address still control who can reach sensitive telemetry.

Development

See CONTRIBUTING.md for the full workflow, including the docs-sync tests and help-screen colorization rules. Run tests and lint locally:

uv run --locked --with pytest --with psutil --with pillow==11.3.0 pytest -q
uv run --locked --with ruff ruff check .
npm ci && npx playwright install --only-shell chromium
npm run test:dashboard
uv run --locked --with build python -m build

Canonical assets use Pillow 11.3.0, the vendored Liberation Mono fonts, and an algorithmic Braille renderer. Their metadata fingerprints the Pillow and FreeType versions as well as the font bytes and renderer source. Regenerate the three canonical asset sets with:

uv run --locked --with pillow==11.3.0 --with psutil python scripts/generate_preview.py --all
uv run --locked --with pillow==11.3.0 --with psutil python scripts/render_gallery.py
uv run --locked --with pillow==11.3.0 --with psutil python scripts/render_showcase.py

To create an explicitly noncanonical preview with local fonts, set MXTOP_PREVIEW_FONT, MXTOP_PREVIEW_BOLD_FONT, and optionally MXTOP_PREVIEW_SYMBOL_FONT, then pass --custom-fonts with a noncanonical output path:

uv run --locked --with pillow==11.3.0 --with psutil python scripts/generate_preview.py \
  --custom-fonts --output /tmp/mxtop-custom.png

More background is available in the usage guide.