Inference Profiler

A profiling tool for LLM inference servers — the perf / PyTorch Profiler equivalent for serving. Point it at any OpenAI-compatible endpoint (vLLM, SGLang, TensorRT-LLM, TGI, llama.cpp server, …) and it measures where time goes, diagnoses bottlenecks, and renders a self-contained HTML report.

Everything runs locally. No cloud backend, no auth, no telemetry.

Why

Engineers usually know latency is bad, but not why. Inference Profiler answers:

• Why is TTFT high — prefill compute, or requests stuck in the scheduler queue?
• Is batching actually working, or is the GPU idle between requests?
• Where is the throughput/latency knee as concurrency grows?
• Are a few long prompts ruining tail latency for everyone?
• Is the server compute-saturated, or starved for work?

Install

pip install inference-profiler          # core
pip install "inference-profiler[gpu]"   # + NVML GPU metrics (NVIDIA)

Quick start

inference-profiler profile \
    --endpoint http://localhost:8000/v1 \
    --model llama3 \
    --requests 500 \
    --concurrency 32 \
    --stream

Console output shows per-stage latency/throughput, findings, and an ASCII waterfall. Full artifacts land in results/:

results/
├── config.json      # exact run configuration (reproducible)
├── metrics.jsonl    # one line per request: timestamps, tokens, errors
├── system.jsonl     # GPU/CPU/RAM/network samples over time
├── summary.json     # aggregated metrics + derived estimates
├── timeline.json    # per-request phase spans (queue/waiting/decode)
├── findings.json    # automatic analysis results with evidence
├── report.html      # self-contained interactive report
└── charts/          # each chart as a standalone HTML file

No GPU? Try the mock server

python examples/mock_server.py --port 8399 --batch-size 8 &
inference-profiler profile --endpoint http://127.0.0.1:8399/v1 --model demo \
    --requests 100 --concurrency 4,8,16,32

The mock server queues requests beyond its batch size — the profiler will find the knee.

What it measures

Measured numbers and heuristic estimates are kept strictly separate — estimates are suffixed *_estimate and their method is documented in docs/methodology.md. Findings are only ever generated from measured data.

Common recipes

# Concurrency sweep: find the throughput/latency knee
inference-profiler profile --endpoint http://localhost:8000/v1 --model llama3 \
    --requests 200 --concurrency 8,16,32,64,128

# Open-loop (Poisson arrivals at 25 req/s) — reveals queueing under overload
inference-profiler profile --endpoint http://localhost:8000/v1 --model llama3 \
    --requests 500 --arrival-mode open --rate 25

# Burst traffic
inference-profiler profile --endpoint http://localhost:8000/v1 --model llama3 \
    --requests 300 --arrival-mode burst --rate 50 --burst-size 40 --burst-idle 5

# Your own prompts (JSONL with a "prompt" field), controlled output lengths
inference-profiler profile --endpoint http://localhost:8000/v1 --model llama3 \
    --dataset examples/prompts.jsonl --output-tokens 64-256 --requests 200

# Everything in a config file (CLI flags override it)
inference-profiler profile --config examples/sweep_config.json

# Re-generate the report from raw data (tweak nothing, re-analyze)
inference-profiler report --results results/

Reading the report

• Findings — ranked observations with the evidence that triggered them, e.g. “GPU utilization averaged only 42%” or “Throughput saturates near concurrency 16”.
• Timeline — a waterfall of every request split into client-queue, waiting (TTFT), and decode phases. Growing gray/yellow bands = queueing; long blue = decode-bound.
• Charts — latency/TTFT histograms, percentile curves, tokens/s over time, in-flight concurrency, GPU utilization & memory, prompt/output length vs latency, and concurrency scaling (on sweeps).

Architecture

cli ─▶ config (pydantic) ─▶ runner ──▶ workload   (synthetic / JSONL prompts)
                              │    ──▶ scheduler  (closed / open-poisson / burst)
                              │    ──▶ collector  (httpx + SSE timing per request)
                              │    ──▶ gpu_monitor (NVML/psutil background sampling)
                              ▼
                    metrics ─▶ timeline ─▶ analysis ─▶ charts ─▶ report.html

See docs/architecture.md. Adding a non-OpenAI backend means implementing one execute_request variant; everything downstream operates on RequestRecords and is backend-agnostic.

Development

git clone https://github.com/SarnadAbhilash/inference_profiler
cd inference-profiler
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytest            # no GPU or server required — uses a mock transport
ruff check .
pre-commit install

License

MIT