brokkr-diagnostics 0.5.2

GPU, storage, thermal & infrastructure diagnostics toolkit

Brokkr Diagnostics

Comprehensive hardware diagnostics for NVIDIA GPU nodes. Collects GPU state, PCIe health, NVLink topology, kernel errors, storage/NVMe health, thermal/IPMI data, and InfiniBand status — structured JSON output, async execution, graceful degradation.

Built for data center operators, HPC administrators, and GPU cluster managers.

Installation

pip install brokkr-diagnostics

Quick Start

# Interactive REPL with tab completion
brokkr-diagnostics --interactive

# Run a specific diagnostic
brokkr-diagnostics --run gpu

# Run all diagnostics
brokkr-diagnostics --run all

Commands

Command	Aliases	Description
driver		NVIDIA driver version & installation checks
gpu		GPU hardware — ECC, retired pages, remapped rows, thermals, power, clocks, PCIe link, processes
nvlink		NVLink status, error counters & topology
lspci	pcie	PCIe topology, link status, AER error counters, ACS & IOMMU
kernel	logs	Kernel log analysis — XID/SXid, storage errors, IOMMU faults, OOM, hung tasks
services		NVIDIA systemd service status
system	proc	System info — /proc files, NUMA topology
storage	nvme, disk	NVMe SMART, error logs, RAID, LVM, filesystem read-only detection
thermal	ipmi, sensors	CPU/board thermals (lm-sensors) & IPMI/BMC sensors + System Event Log
ib		InfiniBand device/port status, error & traffic counters
cuda	cuda-tests	CUDA memory allocation & P2P bandwidth tests
reset		GPU reset sequence (requires root)

Usage Examples

GPU Hardware State

brokkr-diagnostics --run gpu

Returns per-GPU: ECC error counts (volatile + aggregate), retired pages (SBE/DBE/pending), remapped rows (correctable/uncorrectable/pending/failure), memory usage, power draw vs limits, clock speeds, PCIe link gen/width, temperature (GPU + HBM), running processes.

Kernel Log Analysis

brokkr-diagnostics --run kernel

Scans dmesg, journalctl, and log files for errors across domains: NVIDIA XID/SXid errors, NVSwitch failures, storage/NVMe I/O errors, IOMMU page faults, OOM kills, hung tasks, and soft/hard lockups. Each error is categorized by domain and severity.

PCIe Diagnostics

brokkr-diagnostics --run lspci

Enumerates NVIDIA PCIe devices, checks link speed/width degradation, reads AER error counters (correctable, fatal, nonfatal) from sysfs, and checks ACS/IOMMU status for P2P readiness.

Storage & NVMe Health

brokkr-diagnostics --run storage

Collects NVMe SMART data (critical warnings, media errors, spare capacity, wear), NVMe error logs, MD RAID array status, LVM layout, and detects read-only filesystem remounts.

Thermal & IPMI

brokkr-diagnostics --run thermal

Reads CPU/board thermals via lm-sensors and collects IPMI BMC sensor readings (fans, temps, voltages, PSUs) with threshold status. Parses the IPMI System Event Log for hardware fault history.

NVLink Topology

brokkr-diagnostics --run nvlink

Per-GPU NVLink link state, version, remote device type, and error counters (CRC flit, CRC data, replay, recovery). Includes GPU-pair topology mapping.

NVIDIA Driver

brokkr-diagnostics --run driver

Driver version, VBIOS versions, kernel module info (vermagic, parameters), /proc/driver/nvidia state, DKMS logs, installation logs.

InfiniBand

brokkr-diagnostics --run ib

IB device enumeration with firmware/hardware info, per-port state and rate, error counters, and traffic stats from sysfs.

GPU Reset

sudo brokkr-diagnostics --run reset

Kills GPU processes, stops NVIDIA services, unloads kernel modules, performs PCIe bus reset, then reloads everything.

Optional System Tools

These enhance output when available. Missing tools are detected and skipped — never crashes.

Tool	Package	Used by
smartctl	smartmontools	NVMe SMART health
nvme	nvme-cli	NVMe error & smart logs
ipmitool	ipmitool	IPMI sensors & System Event Log
sensors	lm-sensors	CPU/board thermals
lspci	pciutils	PCIe topology & AER counters
mdadm	mdadm	MD RAID status

Requirements

• Python >= 3.10
• Linux
• NVIDIA GPU with drivers loaded (for GPU diagnostics)
• Root/sudo for some operations (GPU reset, IPMI, dmesg)

License

MIT