inflame

Where does your LLM inference latency live?

Feed it vLLM / TGI / Triton request traces; get a self-contained HTML report that shows, per request and fleet-wide, how wall time splits across tokenization → queue → prefill → decode → data io → network. Zero dependencies, stdlib only.

pip install inflame-trace

inflame report traces.jsonl -o report.html   # flamegraph-style HTML
inflame summary traces.jsonl                 # terminal table
inflame formats                              # how to produce each input

The report has three layers:

1. Where latency lives — one icicle bar: share of total wall time per phase.
2. Slowest requests — a waterfall of stacked per-request bars, phases in request order; hover any segment for exact timings, hover the legend to isolate a phase.
3. Phase percentiles — p50/p95/p99 per phase, as a table.

The terminal summary gives you the same phase split as a bar chart you can paste into an incident channel.

Getting traces out of your stack

vLLM inputs are accepted as raw OTLP exports, JSON arrays, or JSONL of spans — whatever your collector writes. Format is auto-detected; force it with --format vllm|tgi|triton.

Reading the report

• queue dominating → batch scheduler is saturated: add replicas, cap concurrency upstream, or shrink max batch tokens.
• prefill dominating → long prompts: cache system prompts (prefix caching), trim context, or split prefill across GPUs.
• decode dominating → normal for long generations; compare decode tok/s against your quantization/kernel expectations — a drop here is where quantization overhead or a mis-tuned kernel shows up.
• data io / network visible at all → payloads are too fat (send token ids, not logits; compress embeddings).

Honest limitations

• TGI does not expose a prefill/decode split; its model compute is one inference phase (completion tokens are estimated from time_per_token).
• vLLM's periodic Avg prompt throughput stats log has no per-request data and is rejected with a pointer to the OTLP setup.
• Triton phases describe the server's view (queue/compute/marshalling); for in-model phase splits use an ensemble that traces per step.

Demo

python3 scripts/make_demo.py   # deterministic 240-request sample -> demo/index.html

Dev

python3 -m pytest -q   # 60+ tests, no network, no deps

MIT.