nvsonar 2.1.2

Active GPU diagnostic tool that identifies performance bottlenecks

NVSonar

GPU monitoring tools show utilization percentages, but this can be misleading. A GPU reporting 100% utilization may actually be computing useful work, or wastefully stalled waiting on memory transfers, thermal throttling, or power limits. NVSonar analyzes real-time patterns from NVML metrics to identify what's actually limiting your GPU performance.

Features

• Bottleneck classification (compute-bound, memory-bandwidth-bound, memory-capacity-bound, power-limited, thermal-throttled, data-starved)
• Temporal pattern detection (clock oscillation, temperature trends, utilization dips, memory leaks)
• Multi-GPU outlier detection via Z-scores
• Health scoring with A-F grades (0-100 per GPU)
• Throttle bitmask decoder with severity levels
• PCIe link degradation and ECC error monitoring
• Actionable recommendations with specific commands
• GPU performance benchmarks (memory bandwidth, compute throughput, PCIe speed)
• Historical tracking with trend analysis over time
• Session monitoring Python API for workload profiling
• JSON and CSV output for automation and scripting

Requirements

• Python 3.10+
• NVIDIA GPU with driver installed
• Linux
• CUDA toolkit (only for nvsonar benchmark, not required for other commands)

Installation and Usage

pip install nvsonar

nvsonar                  # interactive TUI
nvsonar report           # one-shot diagnostic
nvsonar report --json    # structured output for scripts/LLMs
nvsonar report --csv     # CSV output for spreadsheets
nvsonar report --gpu 0   # specific GPU
nvsonar benchmark        # GPU performance benchmarks
nvsonar history          # health trends over time

License

Apache License 2.0

Author

Bekmukhamed Tursunbayev