mprof

A within-kernel profiler for CUDA. Opens the opaque kernel box nsys leaves shut — any kernel, low-overhead — and tells you what hurts and where, as a human view or agent-readable XML. One Rust binary, no Python.

Install

uv tool install mprof

Prebuilt wheels for macOS-arm64 and Linux-x86_64 (else builds from source). view/summary/diff run anywhere on a trace; record needs a CUDA GPU box.

Use

mprof record  -- ./your_program     # capture: CUPTI PC sampling + auto SASS attach
mprof summary trace.mprof           # kernel triage — XML default, also --human / --json
mprof view    trace.mprof           # GUI: diagnosis-first, drill into any kernel
mprof diff    a.mprof b.mprof       # per-kernel timing delta

Views

• Summary — the diagnosis: verdict + per-dimension state (ok/minor/issue) + hot instructions. Facts, not fixes.
• Hotspots — instructions grouped by opcode family, ranked by sample share; click to expand variants.
• Flame — sampled SASS in instruction-category lanes.
• Trace — the outer CUDA timeline; double-click a kernel to drill in.
• Mega / Density — measured per-SM timeline, for .mkprof traces.

PC sampling has no per-sample timestamps, so hotspots are ranked, never placed on a fabricated time axis.

Capture on a GPU box

mprof setup                          # build the CUPTI producer against local CUDA
mprof doctor                         # check toolkit / producer / perf-counter permission
mprof record -- ./your_program       # -> trace.mprof (+ .pc.json sidecar)

doctor prints the exact sudo+reboot step if PC sampling is restricted (NVreg_RestrictProfilingToAdminUsers). Copy the trace anywhere to view it.

For a real within-kernel time axis (PC sampling can't give one), annotate with producers/cuda/mprof.cuh: MPROF_SCOPE(id) around phases, or MPROF_BLOCK(id) at the top of a kernel for a measured SM-occupancy timeline.

Development

cargo build --release
cargo nextest run