mprof 0.1.0

Within-kernel timeline profiler for CUDA: view, summary, diff over .mprof traces

mprof

A within-kernel timeline profiler for CUDA: it opens the opaque kernel box that nsys leaves shut, on a time axis. One Rust binary, no Python runtime.

Install

uv tool install git+https://github.com/Infatoshi/mprof     # from anywhere
uv tool install --force .                                  # from a local checkout

Installing builds the Rust binary, so a Rust toolchain is required on the install machine. (uv tool install mprof by bare name is reserved for a future PyPI release; don't rely on it -- it resolves whatever is on PyPI, not this project. If you hand-delete ~/.local/bin/mprof, reinstall with --force, not a bare reinstall, or uv treats it as a no-op.)

That installs the mprof CLI (a single Rust binary). The analysis commands (view / summary / diff) run anywhere on a captured trace -- this is the local-view half of a remote-record / local-view workflow. Capture (record / setup) needs a CUDA GPU box; see Capture on a GPU box.

From a checkout you can instead cargo build --release and run ./target/release/mprof.

Use

mprof record -- ./your_cuda_program     # capture: CUPTI PC sampling + auto SASS attach
mprof summary trace.mprof               # kernel triage: what hurts, what doesn't
mprof view trace.mprof                  # diagnosis-first GUI; pick kernel/view in-app
mprof diff base.mprof cand.mprof        # per-kernel timing delta (the optimize loop)
mprof doctor                            # check capture prerequisites on this machine

record defaults to CUPTI PC sampling and auto-attaches SASS to the sidecar. summary prints the diagnosis as XML by default (for an LLM agent); add --human for a colored panel or --json for tooling. view opens to the Summary tab; the other views (Hotspots / Flame / Raster / Mega / Density) are in-app tabs. record nvbit|source ... are advanced opt-ins.

Rust Viewer

Open an included sample trace (more in samples/):

mprof samples/cutlass_sm120_pc_gemm.mprof   # PC-sampled GEMM
mprof samples/gemm.mprof                     # source-instrumented timeline

samples/ holds a few small example traces: gemm.mprof (source-instrumented block timeline), gemm_cupti.mprof (CUPTI Activity with v2 launch tags), and cutlass_sm120_pc_gemm.mprof + its .pc.json sidecar (Blackwell GEMM with PC sampling -- the one to try Summary/Hotspots on).

The viewer opens to the Summary diagnosis for the dominant kernel and lets you switch kernel and view in-app -- there are no view-selection launch flags:

mprof trace.mprof                 # opens to the Summary tab
mprof trace.mprof --kernel gelu   # preselect a kernel by name

Top-bar tabs (when a kernel is selected): Summary | Hotspots | Flame | Raster | Mega | Density, plus Trace to return to the outer per-stream/SM timeline.

• Summary -- the diagnosis: verdict + per-dimension state (ok/minor/issue)
  • hot instructions. Same content as mprof summary.
• Hotspots -- instructions ranked by PC-sample share, no fabricated time axis.
• Flame / Raster -- sampled SASS in instruction-category / stall-family lanes.
• Mega / Density -- projected SM-lane views (see the note below).

Interaction (mouse-first):

• Mouse wheel zooms time around the pointer.
• Click-drag the background pans; the horizontal/vertical scrollbars also pan (click a track to jump to that spot).
• Shift + wheel changes row height; Ctrl/Command + wheel does per-mode vertical (row/lane) zoom.
• Double-click a kernel in the Trace timeline to open it.

Mega uses the visual grammar from the MEGAPROFILER layout: GPU-tinted SM bands, separator / loader / consumer / consumer / storer rows, bottom time axis, hover metadata, and progressive detail as you zoom in. With current CUPTI PC-sampling data the SM lanes are projected from sampled PCs, not measured SM residency. Treat it as a readable PC/SASS flame view, not an exact per-SM execution trace.

Use the Density tab in a selected-kernel view to see activity intensity as a per-SM heatmap. MKPROF files use true time-local activity ranges; CUPTI PC captures use projected PC-sample intensity because current PC sidecars do not carry per-sample timestamps.

The Rust viewer also opens MKPROF1.2 files directly:

mprof example_gpu0.mkprof

For .mkprof inputs, Mega renders the real dense activity matrix from the file: GPU/SM lanes, logical instruction types, loader/consumer/storer timing, controller/event markers, and the parameter names carried in the JSON schema.

Open the synthetic ActivityRecord hierarchy demo:

mprof --demo-activity

This demo shows the intended SM-lane activity raster, hover schema, and zoom-revealed instruction slices. In the demo top bar, Rows switches between SM, block, and warp lanes; Depth switches between automatic zoom-based detail, phase-only bars, and instruction bars. It is not captured kernel data.

record defaults to CUPTI PC sampling and auto-attaches SASS to the sidecar (it disassembles the program binary), so a single command gives you the instruction-level data:

mprof record -- ./your_cuda_program     # writes trace.mprof + trace.mprof.pc.json
mprof view trace.mprof

Continuous PC sampling is the default lower-perturbation mode and maps samples back to events by kernel function. SASS attach is best-effort: it is silently skipped when the kernel lives in a shared library (e.g. cuBLAS/cuDNN) rather than the program binary -- for that case attach manually with the advanced mprof sass <trace> <library.so>.

PC sampling gives sampled PC offsets and stall reasons -- the basis for the Summary and Hotspots views. NVBit remains the heavier microscope for dynamic warp/lane traces, active masks, branch paths, and memory addresses.

Double-click a kernel in the Trace timeline to open it; the top-bar tabs (Summary | Hotspots | Flame | Raster | Mega | Density) switch the view of the selected kernel.

In the normal stream timeline, selecting a PC-sampled kernel and zooming until the bar is wide enough draws an inline PC/SASS instruction inset inside that kernel bar. This keeps the outer CUDA stream timeline visible while exposing the inner sampled instruction structure for the selected launch.

Build and capture the reproducible transformer-forward example:

make -C producers/cuda/examples transformer_forward
mprof record -o transformer_forward.mprof -- producers/cuda/examples/transformer_forward
mprof summary transformer_forward.mprof
mprof view transformer_forward.mprof

record auto-attaches SASS (writing transformer_forward.mprof.pc.json), so no separate step is needed.

Diagnose and compare

summary and diff are built into the viewer binary -- no Python runtime:

mprof summary samples/cutlass_sm120_pc_gemm.mprof --human   # triage one kernel
mprof diff baseline.mprof candidate.mprof                   # per-kernel timing delta
cargo nextest run                                                 # tests

diff joins two traces on kernel name and reports the change in average per-launch duration and total GPU time -- the optimize loop is record -> edit kernel -> record -> diff.

Capture on a GPU box

Capture needs a CUDA GPU box: the CUPTI producer is native code built against the local toolkit, and PC sampling needs the perf-counter permission. From a checkout:

mprof setup                              # builds the CUPTI producer against local CUDA
mprof doctor                             # checks toolkit / producer / perf-counter permission
mprof record -- ./your_cuda_program      # capture once the above are green

mprof doctor prints the exact sudo + reboot step if PC sampling is restricted to admin users (NVreg_RestrictProfilingToAdminUsers). For a manual build instead of setup:

make -C producers/cupti
make -C producers/cuda/examples
mprof record -o trace.mprof -- producers/cuda/examples/demo

NVBit support is source-scaffolded, but the NVBit release bundle is not vendored in this repo. Install NVBit separately before building that producer.

Remote / headless GPU boxes

mprof summary is headless, so on a remote GPU server you usually don't move anything -- SSH in and diagnose in place (this is also the agent path):

ssh gpu-box 'mprof summary trace.mprof'          # XML diagnosis, no GUI needed
ssh gpu-box 'mprof summary trace.mprof --human'  # human panel

To open the GUI on another machine, copy the trace with your normal tooling (the .mprof plus its .pc.json sidecar) and view it locally:

scp gpu-box:trace.mprof gpu-box:trace.mprof.pc.json .
mprof view trace.mprof