ingero-annotate 0.1.1

Annotate live Ingero GPU traces with your application's semantics (training step, epoch, inference request_id, model).

ingero-annotate

Add semantic labels to GPU traces captured by Ingero.

What is Ingero?

Ingero is an open source eBPF agent that traces what your GPU is doing in production. It attaches to libcudart, libcuda, and Linux kernel tracepoints, then answers questions like "which CUDA kernels stalled and why," "which process held VRAM," and "which rank in a multi-GPU run was the slow one." No source-code changes, no rebuild. You run ingero trace next to your training or inference job.

What ingero trace cannot know on its own is your application's semantics. It sees kernel launches, memcpy timings, scheduler events. It does not see "this is epoch 3, step 42" or "this is request abc-123 for the llama-3-8b model."

That is what ingero-annotate is for.

What does annotate do?

It is a small Python client that pushes semantic labels into the live trace over a Unix-domain socket. The agent attaches those labels to the trace timeline, so when you later run ingero explain or open the trace in Perfetto, you see kernel events keyed to your concepts.

Training example. Emit a step and phase label at each iteration boundary. A question like "which optimizer step had the 4x backward-pass spike" becomes a one-line query against the trace, instead of squinting at unlabeled kernel timings.

Inference example. Emit request_id and model per request. You can then ask "which incoming request caused the GPU saturation at 14:23:07" and get an answer that names the request, instead of just "something heavy happened."

Install

pip install ingero-annotate

Quick start

Run the agent with the annotation socket bound:

sudo ingero trace --record --annotate

Then from your Python code:

import os
from ingero_annotate import AnnotationWriter

writer = AnnotationWriter()                       # connects, or no-ops silently
writer.write({"step": "42"}, pid=os.getpid())     # one instant annotation
writer.write({"phase": "backward"}, pid=os.getpid())
writer.close()

AnnotationWriter is also a context manager for connection lifecycle:

with AnnotationWriter() as w:
    for step in range(1000):
        w.write({"step": str(step)}, pid=os.getpid())
        train_one_step()

If the agent is not running, or the socket is missing, the writer drops silently. Your training or inference code keeps working unchanged.

Don't want to wire it by hand?

Drop-in adapters for common frameworks live in the agent repo, under examples/integrations/:

• PyTorch Lightning callback
• HuggingFace Trainer callback
• DeepSpeed wrapper
• Accelerate hook
• Ray task hook
• vLLM per-request emitter (inference, emits time-bounded spans)

Each one pulls step / epoch / request_id out of its framework and emits annotations for you.

Wire protocol

Each annotation is one line of NDJSON written to the socket:

{"labels": {"step": "42"}, "pid": 1234, "ts": 1700000000000000000}

labels is required. pid scopes the annotation to one process; without it the label applies trace-wide. ts is optional unix nanoseconds (absent means the agent stamps receive time). The contract also supports time-bounded span annotations (span_start / span_end), used today by the vLLM per-request emitter; this library exposes instant annotations and lets specialized adapters write spans directly.

Label keys and values are validated against the agent contract (charsets, length limits) before being sent.

Per-request inference correlation

For per-request inference workloads, ingero explain --by-request and ingero query --by-request group analyzed events by request_id. The output is a time-overlap slice, not exclusive kernel ownership: under continuous batching a single kernel can belong to many in-flight requests. The agent prints that caveat with every per-request report. Full semantics, trust-domain scope, and example output: docs/commands.md.

Links

• Agent: https://github.com/ingero-io/ingero
• Docs: https://github.com/ingero-io/ingero/blob/main/docs/commands.md
• Issues: https://github.com/ingero-io/ingero/issues

License

Apache-2.0.