Skip to main content

xprobe

Find out why your CPU, Python, or CUDA process is slow — in two commands.

A bounded Linux profiler for AI coding agents and the humans working with them:
eBPF + perf + NVIDIA CUPTI, one CLI, strict JSON, no daemon.

CI status Latest release Apache-2.0 license CUDA 12.x and 13.x

xprobe triages a slow PyTorch CUDA process into ranked findings, then measures request to launch 1.14 ms, launch to kernel 99 µs, and kernel 1.2 µs on CPU and GPU timelines

Real numbers from the v0.6.0 release gate on an RTX 3060: a PyTorch CUDA workload and a request → launch → kernel path.

What it does

Point xprobe at a running process. It watches one bounded window and tells you where the time went — on-CPU hotspots, threads waiting for a CPU or blocked, container CPU throttling, hot syscalls, GPU kernels, and GPU clock limits — ranked by impact, each with a ready next measurement. The second command then measures that exact latency: a function, a syscall, a CUDA API call, a kernel, a copy, an NVTX range, or a whole request → launch → kernel path from a single capture.

  • Built for AI agents. Ships as an Agent Skill for Claude Code, Codex, and Cursor. Compact JSON with a good/caution quality verdict, so an agent can act on results without reading 100 KB of events.
  • CPU, Python, and GPU in one tool. Sampled native and CPython stacks, eBPF uprobes, syscalls, tracepoints, CPython GC, and CUDA 12/13 activity via CUPTI, correlated with explicit confidence.
  • Bounded by design. Every capture is limited by samples or duration, timeout, and capacity; there is no daemon; CUDA injection is disclosed before it happens; the raw events are always kept for offline re-analysis.
  • Honest about what it knows. Sampled shares are estimates, temporal matches are labeled heuristic, and anything a capture cannot answer is listed as not covered.

Install

Install the version-matched Skill with the open Agent Skills CLI. This is the only installation action required from the user:

npx skills@1 add \
  https://github.com/itdevwu/xprobe/tree/v0.6.3/skills/xprobe-measure-latency \
  --global

The installer detects Codex, Claude Code, Cursor, and other compatible agents. When invoked, the Skill checks for the matching xprobe CLI and installs or repairs it under a writable prefix before profiling. It can then diagnose and adjust path, permission, NVIDIA, CUDA, or CUPTI problems from live evidence. Node.js is only needed for Skill installation, not for xprobe itself.

To install only the CLI, use pip or the release archive:

pip install xprobe

See Installation for the release archive, SPDX SBOMs, and GitHub-hosted build attestation verification.

Measure

Triage an unknown process over one representative window. Findings are ranked by their estimated share of the window, and each carries a ready next step:

xprobe measure --pid 4242 --triage --duration-ms 2000 > triage.json
jq '.findings[0].next.spec' triage.json > next.json
xprobe measure --spec next.json

Measure a known boundary directly, or several related boundaries from one capture with a measurement graph:

xprobe measure --pid 4242 \
  --from 'cuda:launch_api:exit' \
  --to 'cuda:kernel_start:name~flash.*' \
  --match exact --samples 100

Output is JSON whenever stdout is not a terminal. Results carry latency statistics, a good/caution quality verdict, rank exemplar pairs, and the path of the bounded event capture, which measure --input re-correlates offline. See the CLI contract for every mode.

Public CLI

Command Purpose
doctor Report local eBPF, ptrace, NVIDIA, CUDA, and CUPTI capabilities
discover List NVML-confirmed CUDA context holders under a process-tree root
validate Check an event pair, graph, or inventory mode without attaching
measure Triage, inventory, or correlate bounded live or recorded evidence

measure --pid automatically loads the matching CUDA 12 or CUDA 13 CUPTI Agent when a selected endpoint requires it. It reports the target mutation on stderr and in JSON, disables collection afterward, and leaves the shared object mapped for safe reactivation. NVTX ranges are the exception: set NVTX_INJECTION64_PATH before the target's first NVTX call because online attach cannot retrofit an initialized NVTX dispatch.

Support

Surface Current support
OS/architecture Linux x86_64, glibc 2.34 or newer
Triage One window of thread time, cgroup throttling, CPU samples, syscalls, NVML health, and CUDA kernels
Measurement graphs Up to eight edges from one capture
CPU inventory PID-scoped sampled user stacks with native ELF and CPython perf-map symbols
Host events ELF functions, named syscall/tracepoint boundaries, and CPython GC USDT
Coarse aggregation Bounded syscall lifecycle and CUDA kernel/memcpy/memset groups, with pageable/pinned copies
CUDA callbacks Runtime and Driver API entry/exit
GPU activity Kernel, memcpy, and memset start/end
Application ranges Bounded ASCII NVTX thread and process ranges
CUDA/CUPTI 12.x and 13.x with automatic major selection
Correlation exact, first-after, nearest, stack-nested, stream-order
Online injection same mount namespace; ptrace permission required

GPU-to-GPU durations remain available across the supported majors. Cross CPU/GPU subtraction requires the Agent's runtime alignment check to pass; otherwise measure returns CLOCK_ALIGNMENT_FAILED rather than treating an offset CUPTI clock as host monotonic.

Documentation

Source builds and hardware tests are documented under Development. Implemented behavior is defined by code, tests, and schemas. Licensed under Apache-2.0.

Metadata

Release files for xprobe 0.6.3

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distribution (wheel)

Table of built distributions (wheels) for xprobe 0.6.3
File Interpreter ABI Platform Actions
xprobe-0.6.3-py3-none-manylinux_2_34_x86_64.whl Python 3 none Linux glibc 2.34+ x86-64 Details for xprobe-0.6.3-py3-none-manylinux_2_34_x86_64.whl

Release files / xprobe-0.6.3-py3-none-manylinux_2_34_x86_64.whl

Download URL xprobe-0.6.3-py3-none-manylinux_2_34_x86_64.whl
Size 2.7 MB
Tags Linux glibc 2.34+ x86-64 Python 3
SHA-256 checksum
How to use checksums
c475fff4fc7d53a546fecdd6072c6591b88c197ceab952ef2411601bf19ce1d2
BLAKE2b-256 checksum
How to use checksums
2da208d28e69e1f69189af919023e75c7c6fc729f39bcadafad47f4208cf0611
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.

Transparency log

Security for xprobe 0.6.3

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

1 file in this release has been uploaded with provenance metadata.

Attestation for xprobe-0.6.3-py3-none-manylinux_2_34_x86_64.whl

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 21, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.6.3 This release

1 release file Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page