brokkr-diagnostics 0.6.0

GPU, storage, thermal & infrastructure diagnostics toolkit

Brokkr Diagnostics

Comprehensive hardware diagnostics for NVIDIA GPU nodes. Collects GPU state, PCIe health, NVLink topology, kernel errors, storage/NVMe health, thermal/IPMI data, and InfiniBand status — structured JSON output, async execution, graceful degradation.

Built for data center operators, HPC administrators, and GPU cluster managers.

Installation

pip install brokkr-diagnostics

Quick Start

# Interactive REPL with tab completion
brokkr-diagnostics --interactive

# Run a specific diagnostic
brokkr-diagnostics --run gpu

# Run all diagnostics
brokkr-diagnostics --run all

# Run and send all diagnostics to Brokkr
brokkr-diagnostics --send

# Run and send a specific diagnostic to Brokkr
brokkr-diagnostics --send gpu

Commands

Command	Aliases	Description
driver		NVIDIA driver version & installation checks
gpu		GPU hardware — ECC, retired pages, remapped rows, thermals, power, clocks, PCIe link, processes
nvlink		NVLink status, error counters & topology
lspci	pcie	PCIe topology, link status, AER error counters, ACS & IOMMU
kernel	logs	Kernel log analysis — XID/SXid, storage errors, IOMMU faults, OOM, hung tasks
services		NVIDIA systemd service status
system	proc	System info — /proc files, NUMA topology
storage	nvme, disk	NVMe SMART, error logs, RAID, LVM, filesystem read-only detection
thermal	ipmi, sensors	CPU/board thermals (lm-sensors) & IPMI/BMC sensors + System Event Log
ib		InfiniBand device/port status, error & traffic counters
cuda	cuda-tests	CUDA memory allocation & P2P bandwidth tests
reset		GPU reset sequence (requires root)

Usage Examples

GPU Hardware State

brokkr-diagnostics --run gpu

Returns per-GPU: ECC error counts (volatile + aggregate), retired pages (SBE/DBE/pending), remapped rows (correctable/uncorrectable/pending/failure), memory usage, power draw vs limits, clock speeds, PCIe link gen/width, temperature (GPU + HBM), running processes.

Kernel Log Analysis

brokkr-diagnostics --run kernel

Scans dmesg, journalctl, and log files for errors across domains: NVIDIA XID/SXid errors, NVSwitch failures, storage/NVMe I/O errors, IOMMU page faults, OOM kills, hung tasks, and soft/hard lockups. Each error is categorized by domain and severity.

PCIe Diagnostics

brokkr-diagnostics --run lspci

Enumerates NVIDIA PCIe devices, checks link speed/width degradation, reads AER error counters (correctable, fatal, nonfatal) from sysfs, and checks ACS/IOMMU status for P2P readiness.

Storage & NVMe Health

brokkr-diagnostics --run storage

Collects NVMe SMART data (critical warnings, media errors, spare capacity, wear), NVMe error logs, MD RAID array status, LVM layout, and detects read-only filesystem remounts.

Thermal & IPMI

brokkr-diagnostics --run thermal

Reads CPU/board thermals via lm-sensors and collects IPMI BMC sensor readings (fans, temps, voltages, PSUs) with threshold status. Parses the IPMI System Event Log for hardware fault history.

NVLink Topology

brokkr-diagnostics --run nvlink

Per-GPU NVLink link state, version, remote device type, and error counters (CRC flit, CRC data, replay, recovery). Includes GPU-pair topology mapping.

NVIDIA Driver

brokkr-diagnostics --run driver

Driver version, VBIOS versions, kernel module info (vermagic, parameters), /proc/driver/nvidia state, DKMS logs, installation logs.

InfiniBand

brokkr-diagnostics --run ib

IB device enumeration with firmware/hardware info, per-port state and rate, error counters, and traffic stats from sysfs.

GPU Reset

sudo brokkr-diagnostics --run reset

Kills GPU processes, stops NVIDIA services, unloads kernel modules, performs PCIe bus reset, then reloads everything.

Sending Diagnostics to Brokkr

The --send (-s) flag runs diagnostics and POSTs the results to the Brokkr API. This is used for remote monitoring and automated health reporting.

# Send all diagnostics
brokkr-diagnostics --send

# Send a specific diagnostic
brokkr-diagnostics --send gpu

When sending all diagnostics, commands that require root or are synchronous-only (e.g., reset) are automatically skipped.

Authentication

Credentials are resolved in this order:

1. Creds file — /var/lib/brokkr/phone-home-creds.json (keys: cipher, signature)
2. Fallback script — Parsed from /var/lib/cloud/scripts/per-boot/90-phone-home.sh if the creds file is missing or incomplete
3. Environment variables — BROKKR_AUTH_CIPHER and BROKKR_AUTH_SIGNATURE override any file-based values

Configuration

Environment Variable	Default	Description
BROKKR_API_URL	https://brokkr.hydrahost.com/api/v1	API base URL
BROKKR_AUTH_CIPHER	(from creds file)	Authentication cipher
BROKKR_AUTH_SIGNATURE	(from creds file)	Authentication signature
BROKKR_TIMEOUT	30	HTTP request timeout in seconds

Optional System Tools

These enhance output when available. Missing tools are detected and skipped — never crashes.

Tool	Package	Used by
smartctl	smartmontools	NVMe SMART health
nvme	nvme-cli	NVMe error & smart logs
ipmitool	ipmitool	IPMI sensors & System Event Log
sensors	lm-sensors	CPU/board thermals
lspci	pciutils	PCIe topology & AER counters
mdadm	mdadm	MD RAID status

Requirements

• Python >= 3.10
• Linux
• NVIDIA GPU with drivers loaded (for GPU diagnostics)
• Root/sudo for some operations (GPU reset, IPMI, dmesg)

License

MIT