brokkr-diagnostics 0.3.9

NVIDIA GPU and InfiniBand diagnostics toolkit

Brokkr Diagnostics

A comprehensive diagnostic toolkit for NVIDIA GPU and InfiniBand infrastructure. Designed for data center operators, HPC administrators, and GPU cluster managers who need deep visibility into hardware health and configuration.

Features

• NVIDIA Driver Diagnostics - Version checks, module loading, ABI compatibility, installation logs
• GPU Hardware Analysis - GPU identification, memory, clocks, power states, thermal monitoring
• CUDA Environment - CUDA runtime, compute capability, library versions
• PCIe Topology - Bus enumeration, link speed/width validation, topology mapping
• Kernel Monitoring - SystemD services, kernel logs, error detection, dmesg analysis
• System Information - NUMA topology, /proc filesystem analysis, memory configuration
• InfiniBand Diagnostics - IB device enumeration, port status, link state
• GPU Reset - Safe GPU reset functionality for stuck or hung devices
• Confidential Computing - Enable/disable NVIDIA Confidential Compute mode (requires root)

Installation

pip install brokkr-diagnostics

Usage

Interactive Mode

Launch the interactive menu to explore diagnostics:

brokkr-diagnostics

Command-Line Mode

Run specific diagnostics non-interactively:

# Check NVIDIA driver installation and versions
brokkr-diagnostics --run driver

# Analyze GPU hardware state
brokkr-diagnostics --run gpu

# Inspect PCIe topology
brokkr-diagnostics --run lspci

# Check SystemD services
brokkr-diagnostics --run services

# Scan kernel logs for errors
brokkr-diagnostics --run kernel

# View system information (NUMA, /proc)
brokkr-diagnostics --run system

# InfiniBand diagnostics
brokkr-diagnostics --run ib

# CUDA environment checks
brokkr-diagnostics --run cuda

# Run all diagnostics
brokkr-diagnostics --run all

# Reset all GPUs (requires appropriate permissions)
brokkr-diagnostics --run reset

# Enable Confidential Compute mode (requires root)
sudo brokkr-diagnostics --run "cc on"

# Disable Confidential Compute mode (requires root)
sudo brokkr-diagnostics --run "cc off"

Diagnostic Modules

Driver Diagnostics (driver)

Validates NVIDIA driver installation and configuration:

• nvidia-smi version and availability
• Driver version from /proc/driver/nvidia and nvidia-smi
• Kernel module version magic (vermagic) for ABI compatibility
• Installation/uninstallation logs
• DKMS compilation logs
• Loaded modules status (nvidia, nvidia_drm, nvidia_modeset, nvidia_uvm, nvidia_peermem)
• Module parameters and configuration

GPU Hardware (gpu)

Comprehensive GPU hardware state analysis:

• GPU enumeration and identification
• Memory capacity and utilization
• Clock speeds (graphics, memory, SM)
• Power state and consumption
• Temperature monitoring
• Fan speed
• PCIe generation and link width
• Compute mode
• Persistence mode
• ECC status and errors
• Performance state (P-state)

PCIe Diagnostics (lspci)

PCIe bus topology and link analysis:

• Device enumeration on PCIe bus
• Link speed and width (current vs. maximum)
• Bus topology mapping
• Device class identification
• Vendor and device IDs

Service Status (services)

SystemD service monitoring:

• NVIDIA persistence daemon
• Fabric manager
• GPU driver services
• Service health and uptime

Kernel Logs (kernel)

Kernel message analysis:

• Recent kernel logs (dmesg)
• NVIDIA-related messages
• Error detection and filtering
• Hardware error events
• PCIe link errors
• GPU initialization logs

System Information (system)

Linux system configuration:

• NUMA topology
• CPU information from /proc/cpuinfo
• Memory configuration from /proc/meminfo
• Kernel version from /proc/version
• System uptime

InfiniBand (ib)

InfiniBand network diagnostics:

• IB device enumeration
• Port status and state
• Link layer information
• Physical state
• Rate and link speed

CUDA Diagnostics (cuda)

CUDA environment validation:

• CUDA runtime version
• Compute capability per GPU
• CUDA library paths
• nvcc compiler availability
• CUDA sample compilation tests

Requirements

• Python >= 3.8
• Linux operating system (Ubuntu, CentOS, RHEL, etc.)
• NVIDIA GPU (for GPU-related diagnostics)
• Root/sudo access (for some operations like GPU reset and Confidential Compute)

Dependencies

• numpy>=1.24.4
• nvidia-gpu-admin-tools>=2025.11.21
• termcolor>=2.4.0
• prompt_toolkit>=3.0.0

Use Cases

Data Center Operations

• Pre-deployment hardware validation
• Post-maintenance verification
• Troubleshooting GPU issues
• Configuration auditing

HPC Clusters

• Node health checks
• GPU allocation validation
• Performance baseline verification
• InfiniBand network validation

ML/AI Infrastructure

• Training environment validation
• Multi-GPU setup verification
• CUDA environment checks
• Memory and thermal monitoring

CI/CD Pipelines

• Automated hardware testing
• Regression detection
• Configuration compliance

License

MIT License

Author

Brokkr by Hydra Host Contact: [e-mail]

Contributing

Issues and pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.