---

nsys-ai reads .nsys-rep, .parquetdir, or .sqlite exports from NVIDIA Nsight Systems and turns them into something you can navigate and reason about: a web timeline, terminal viewers, a before/after diff that reports whether a change actually helped, and a set of deterministic analysis skills an LLM agent can drive. .nsys-rep files use a Parquet cache by default; .sqlite remains the compatibility path.

Installation

pip install nsys-ai

No CUDA and no Nsight install are required to analyze a profile. Python 3.10+ only. (Capturing a new .nsys-rep, or converting one, needs the nsys CLI on your machine; analyzing an existing .sqlite does not.)

Quick start

1. Capture a profile

For ML training, capture a few representative iterations rather than the whole run — it keeps the profile small and the profiler overhead low. Mark the region with the CUDA profiler API and trace CUDA plus NVTX:

import torch

for step in range(warmup):
    train_step()
torch.cuda.synchronize()
torch.cuda.cudart().cudaProfilerStart()
for step in range(3):            # profile these iterations
    train_step()
torch.cuda.synchronize()
torch.cuda.cudart().cudaProfilerStop()

nsys profile --capture-range=cudaProfilerApi --trace=cuda,nvtx \
  -o my_training python train.py
# -> my_training.nsys-rep

--trace=cuda is what every skill relies on (GPU kernels, memory copies, CUDA API). nvtx adds the annotation hierarchy that drives the iteration, region, and layer views. To use the iteration tools (iters, diff --iteration), annotate each step with a consistent NVTX marker — see Focused Profiling and NVTX Annotations.

No workload handy? Download an example profile:

cd examples/example-20-megatron-distca && python download_data.py
# -> output/megatron_distca.nsys-rep

2. Open it

# Default: open the web timeline in your browser
nsys-ai my_training.nsys-rep

# Metadata and GPU info
nsys-ai info my_training.nsys-rep

# GPU kernel summary
nsys-ai summary my_training.nsys-rep --gpu 0

Prefer the terminal? The TUIs work the same way:

nsys-ai timeline my_training.nsys-rep --gpu 0   # Perfetto-style horizontal timeline
nsys-ai tui my_training.nsys-rep --gpu 0        # NVTX tree browser

3. Compare two runs

nsys-ai diff before.sqlite after.sqlite

4. Keep the investigation in one session

The 0.3.0 command surface is built around a session directory: findings, proposals, run specifications, diffs, and decisions can move between CLI, Web, TUI, and MCP without reconstructing state from terminal output.

SESSION=/tmp/nsys-ai/run-001

nsys-ai doctor run-before/profile.sqlite --format json
nsys-ai diagnose run-before/profile.sqlite --session "$SESSION"
nsys-ai ask --session "$SESSION" "what is the main bottleneck?"
nsys-ai diff run-before/profile.sqlite run-after/profile.sqlite \
  --no-ai --session "$SESSION"
nsys-ai review --session "$SESSION"

For a complete diagnose → propose → re-profile → diff → decision walkthrough, including the RunSpec required by propose, see the user guide.

Web timeline

A browser-based multi-GPU viewer with progressive rendering — no --trim required. This is the default view when you run nsys-ai <profile>.

nsys-ai my_training.nsys-rep                       # opens in your browser
nsys-ai timeline-web my_training.nsys-rep --gpu 0 1 2 3

• Multi-GPU stacked view with color-coded separators
• Progressive rendering — pre-builds the NVTX tree at startup, then serves tiles in about a millisecond each
• NVTX hierarchy bars (L0-L5) per GPU
• AI chat sidebar (press a) and kernel search (press /)

Input	Action
Swipe / h l / arrows	Pan through time
Swipe up-down / j k	Select stream
Pinch / Shift+scroll / + -	Zoom
f or 0	Fit full time range
Tab	Next kernel
/	Search kernels
n	Toggle NVTX
a	AI chat
?	Help overlay

Timeline TUI

A Perfetto-style horizontal viewer with per-stream kernels, NVTX hierarchy bars, and a time-cursor navigation model.

Key	Action
arrows	Pan time / select stream
Shift+arrows	Page pan (quarter viewport)
Tab	Snap to next kernel
+ -	Zoom
/	Filter kernels by name
m	Minimum-duration threshold
d	Toggle demangled names
B	Save bookmark (with kernel + NVTX context)
C	Config panel (stream rows, tick density, NVTX depth)
h	Full help overlay

Profile diff

Comparing two profiles is the point of nsys-ai: it reports not just what changed but whether the change is a likely regression or improvement.

# Terminal report
nsys-ai diff before.sqlite after.sqlite

# Interactive side-by-side web comparison
nsys-ai diff-web before.sqlite after.sqlite

# A specific device or time window
nsys-ai diff before.sqlite after.sqlite --gpu 0 --trim 39 42

# Compare one aligned iteration
nsys-ai diff before.sqlite after.sqlite --iteration 0

# Markdown (for a PR or issue) or JSON (for scripting)
nsys-ai diff before.sqlite after.sqlite --format markdown -o diff.md
nsys-ai diff before.sqlite after.sqlite --format json

# Gate CI: exit non-zero when the verdict is a likely regression, or when the
# two profiles could not be compared at all (for example one side recorded no
# GPU kernel activity because the profiling step failed)
nsys-ai diff before.sqlite after.sqlite --exit-on-regression

# Same gate with a custom regression threshold (default 5%)
nsys-ai diff before.sqlite after.sqlite --gate 3.0

The report covers top regressions and improvements, new and removed kernels, NVTX region deltas, compute/NCCL overlap and idle changes, and a step-time category rollup (compute / communication / idle). With --format json it adds a top-level verdict, a comparability_confidence score, and a stable content-derived profile_id per side. With no --gpu, the diff aggregates across every device.

Flag	Default	Description
--gpu N	all GPUs	Restrict to one device
--trim START END	full span	Compare only this window (seconds)
--iteration N	—	Compare one aligned iteration (needs an NVTX marker)
--format	terminal	terminal | markdown | json
--limit N	15	Top regressions/improvements to show
--sort	delta	delta | percent | total
--exit-on-regression	—	Exit 1 when the verdict is regression_likely, or inconclusive because the profiles could not be compared
--gate PCT	5.0	Regression threshold (%) for the verdict; implies --exit-on-regression

Baselines

A diff needs something to compare against. Passing a raw file path is fragile in CI: the path drifts between jobs and the file may not survive to the next run. The baseline command keeps a local store of named snapshots so a stable name, not a path, resolves the comparison.

# Tag a known-good run under a name (copies the resolved .sqlite into the store)
nsys-ai baseline tag main run.sqlite --reason "green main @ abc123"

# List and inspect what has been tagged
nsys-ai baseline list
nsys-ai baseline show main

# Compare a candidate against a tagged baseline by name
nsys-ai diff --against baseline:main candidate.sqlite

tag resolves the profile (including a .nsys-rep sidecar), copies the self-contained .sqlite into the store, and records a deterministic meta.json (content-derived profile_id, source path, reason, tagger, timestamp). The snapshot stays valid even if the original file moves.

The store lives in .nsys-ai-baselines/ under the current directory. Set NSYS_AI_BASELINE_ROOT to point tag and resolve at a shared location so a job that tags and a later job that diffs find the same store regardless of CWD:

export NSYS_AI_BASELINE_ROOT="$CI_CACHE/nsys-baselines"
nsys-ai baseline tag main run.sqlite --reason "green main" \
  && nsys-ai diff --against baseline:main candidate.sqlite

The baseline:<name> reference is accepted anywhere a baseline profile path is, via --against or as the before positional.

Commands

Command	Description
open	Quick-open a profile in the web UI or TUI
timeline-web	Web multi-GPU timeline (progressive rendering)
timeline	Timeline TUI
tui	NVTX tree TUI
web	Web viewer server
info	Profile metadata and GPU hardware
doctor	Check environment, ingest, cache, and profile health
profile	Capture a workload and write a reproducible RunSpec
warm	Build the Parquet cache and NVTX kernel map up front
summary	Top kernels and stream breakdown
analyze	Full auto-report (--format json emits evidence findings)
overlap	Compute / NCCL overlap analysis
nccl	NCCL collective breakdown
iters	Auto-detect training iterations
tree / markdown	NVTX hierarchy as text / markdown
search	Search kernels and NVTX by name
report	Generate a performance report
diff	Before/after profile comparison
diff-web	Side-by-side comparison web viewer
baseline	Manage named baseline snapshots (tag, list, show)
diagnose	Run the default evidence pack and publish findings
propose	Turn one finding into a verifiable proposal
review	Compare a pair or resume a session decision path
optimize	Run diagnose → propose → re-profile → diff as one session
chat	AI chat TUI for a profile
ask	One-shot AI question about a profile
agent	Agent auto-analysis (analyze, ask)
skill	List and run analysis skills
evidence	Build evidence findings for the timeline overlay
root-cause	Browse and submit root-cause patterns
cutracer	Instruction-level drill-down (check, install, plan, run, analyze)
export / export-csv / export-json	Perfetto JSON, flat CSV, flat JSON
viewer / timeline-html	Interactive HTML report / timeline

Run nsys-ai <command> --help for flags.

Analysis cache

The first command run against a profile builds a <profile>.nsys-cache directory next to it: the tables analysis needs, exported to Parquet and queried through DuckDB. Later commands reuse it and open in well under a second. The cache is rebuilt automatically when the profile changes; deleting the directory is safe.

Measured on the reference captures (12 cores, 15 GB RAM), running eight skills: top_kernels, gpu_idle_gaps, overlap_breakdown, memory_transfers, kernel_launch_overhead, stream_concurrency, tensor_core_usage, nvtx_layer_breakdown. Reproduce with python scripts/bench_cache.py <profile> --basket auto-policy.

Profile	Build	First NVTX query	Cache size	Eight skills: direct query → cached query
93 MB	1.9 s	+3.7 s	17 MB	5.8 s → 0.6 s
235 MB	2.6 s	+3.4 s	22 MB	11.2 s → 3.7 s
924 MB	8.4 s	+11.2 s	84 MB	33.2 s → 8.5 s
3.7 GB	27.4 s	+49.7 s	277 MB	83.4 s → 16.3 s

The build exports Parquet; the kernel-to-NVTX map is built separately by the first query that needs it, which is the "first NVTX query" column. Both are paid once per profile.

On this workload the first run is never slower end to end than querying the export directly, and lighter on memory at all four sizes, so the build is the default — "this workload" being the eight skills above; a one-shot is a different trade, see below. The middle two sizes are a clear win on time (23% and 26%); at 93 MB and 3.7 GB the time difference is 2-5%, inside run-to-run variance, and what those sizes gain is memory — 7.1 GB against 10.2 GB on the largest. Every command after the first is the warm row, which is where the cache pays for itself outright.

Two cases where it is not what you want:

• One command against a very large profile, and no follow-up. Set NSYS_AI_CACHE_MODE=direct to query the SQLite export in place — instant start, slower queries, and no cache written. nsys-ai skill run also takes --no-cache for the same effect. A light one-shot workload is also the case where the build costs more memory than it saves, so prefer direct if you are tight on RAM.
• A read-only or full disk. No setting needed: nsys-ai checks before it builds, says why it declined, and queries the export directly.

NSYS_AI_CACHE_MODE=parquet forces the build in the other direction. Both values only decide whether a cache gets built: an existing valid cache is still used, so delete the directory if you want the export read in place.

Skills

Skills are self-contained analysis units that run without an LLM. The packaged registry covers kernels, memory, NCCL/communicators, NVTX, MFU, idle, root-cause, profile health, and more. Run skill list for the live catalog; the count is intentionally not a compatibility contract.

nsys-ai skill list                                 # full catalog
nsys-ai skill run top_kernels profile.sqlite
nsys-ai skill run nccl_breakdown profile.sqlite
nsys-ai skill run profile_health_manifest profile.sqlite --format json

A few common ones:

Skill	What it does
top_kernels	Heaviest GPU kernels by total time
gpu_idle_gaps	Pipeline bubbles between kernels
memory_transfers	H2D / D2H / D2D transfer breakdown
nccl_breakdown	NCCL collective summary by type
nccl_communicator_analysis	Per-communicator NCCL topology and efficiency
overlap_breakdown	Compute / communication overlap
kernel_launch_overhead	CPU-to-GPU dispatch latency
region_mfu	Model FLOPs utilization for an NVTX region
profile_health_manifest	One-shot health summary (run this first)

Skills are extensible — add one by dropping a Python file that exports a SKILL constant. See skill list for the full catalog.

AI analysis (optional)

The agent is a CUDA performance expert that runs the skills and cites the evidence — kernel names, durations, timestamps — behind each diagnosis rather than guessing. Targeted ask answers use a fixed evidence-first shape: summary, primary diagnosis, cited evidence, confidence, recommended action, and a final runnable verification command.

nsys-ai agent analyze profile.sqlite
nsys-ai agent ask profile.sqlite "why are there bubbles in the pipeline?"
nsys-ai ask profile.sqlite "is NCCL overlapping with compute?"
nsys-ai chat profile.sqlite                        # interactive chat TUI

The AI features need a provider API key. Set one of:

export ANTHROPIC_API_KEY=...      # or
export OPENAI_API_KEY=...         # or
export GEMINI_API_KEY=...
export NSYS_AI_MODEL=...          # optional: pick a specific model

Install the dependencies with the agent extra:

pip install 'nsys-ai[agent]'

With a key, targeted ask uses the model to select deep-dive skills and synthesize the Summary; the remaining evidence-first sections and verification command are built deterministically from skill output. If no key is set, the agent returns the same answer shape with a deterministic Summary.

Claude Code plugin

nsys-ai ships as a Claude Code plugin: the /nsys-ai slash command turns a profile into a root cause, a proposed fix, and an annotated timeline. See docs/claude-plugin-quickstart.md to install and docs/claude-plugin.md for the full reference.

Documentation

Start with the User guide — one workload from capture to a recorded decision, on the command line. To drive the same workflow in a browser instead, see Guided loop setup.

Useful entry points for the next question:

• upgrading from 0.2.3 → Migrating to 0.3.0
• something is not working → Troubleshooting
• checking an input before analysis → Profile inputs
• choosing a browser surface → Choosing a Web viewer
• running a focused, no-LLM analysis → Analysis skills

The complete, maintained documentation index is docs/README.md.

The rest of the docs/ directory mirrors the relevant NVIDIA Nsight Systems reference (capture, schema, NVTX, CUDA/NCCL trace) plus nsys-ai project guides:

Guide	Topic
User guide	Capture, diagnose, propose, diff, decide — end to end
doctor	Environment and profile health checks
NVIDIA nsys CLI	The upstream nsys profiler CLI (capture-time)
SQLite schema	Nsight export tables and queries
NVTX annotations	Annotating your code (and iteration markers)
CUDA trace	GPU kernel and memory tracing
NCCL tracing	Multi-GPU collective analysis
Python / PyTorch	Profiling PyTorch workloads
Containers	Profiling inside Docker / Slurm
Focused profiling	Capturing representative iterations
CUTracer	Instruction-level drill-down for top kernels

The docs/sqlite-explorer/ directory holds an interactive HTML explorer for the Nsight SQLite schema — open docs/sqlite-explorer/index.html in a browser.

Install tiers

pip install nsys-ai              # core: CLI, TUIs, skills, web/diff viewers
pip install 'nsys-ai[agent]'     # + LLM-backed agent (anthropic + litellm)
pip install 'nsys-ai[chat]'      # + chat TUI
pip install 'nsys-ai[mcp]'       # + stdio MCP transport (`nsys-ai-mcp`)
pip install 'nsys-ai[cutracer]'  # + CUTracer instruction-level workflow
pip install 'nsys-ai[all]'       # everything

The ai extra is kept as an alias of agent for backward compatibility.

Development

git clone https://github.com/GindaChen/nsys-ai.git
cd nsys-ai
pip install -e '.[dev]'
pytest tests/ -v

See CONTRIBUTING.md for the full contributor workflow, test layers, skill walkthrough, fixture policy, and pull-request checklist.

Guided optimization loop (diagnose → propose → re-profile → diff → accept): see the User guide for the CLI path and docs/guided-loop-setup.md for the browser path.

---

License

MIT — see LICENSE.