inference-autopsy 0.1.0

Trace-first profiler and regression tester for AI inference systems.

Inference Autopsy

Who killed my TTFT?

Inference Autopsy is an open-source black-box profiler, workload replayer, and regression tester for OpenAI-compatible LLM inference endpoints.

It records request-level and token-level traces, measures TTFT, ITL, tail latency, throughput, streaming stalls, and error rates, then turns those measurements into reproducible reports, baseline diffs, and CI regression gates.

Your LLM endpoint got slow. We found the body in the token stream.

The goal is a polished local CLI plus static HTML reports, not a hosted SaaS dashboard.

Public Demo

• Live project page: https://kaseyho.github.io/Inference-Autopsy/
• Sample report: https://kaseyho.github.io/Inference-Autopsy/sample-report.html
• PyPI package: https://pypi.org/project/inference-autopsy/

The hosted report is generated from synthetic traces, so it is safe to share publicly and does not expose private prompts, API keys, or endpoints.

Install

pip install inference-autopsy

For local development:

python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
pytest
ruff check .

Why This Exists

LLM inference latency is not just one number.

A request can be slow because of first-token delay, slow decode, long prompts, tail latency, rate limits, stream stalls, output bloat, or concurrency collapse. Aggregate benchmark numbers can tell you that something changed. Inference Autopsy is designed to help answer:

1. How fast is this endpoint?
2. Why is it slow?
3. Can I reproduce the workload?
4. Did my model or deployment regress?
5. Can I explain the failure in one memorable line?

The wedge is:

benchmark -> trace -> diagnose -> report -> replay -> regression gate

Existing tools such as NVIDIA GenAI-Perf, GuideLLM, and LLMPerf measure important LLM serving metrics. Inference Autopsy focuses on trace-level reproducibility, diagnosis, human-readable reports, and CI gates for OpenAI-compatible endpoints.

What It Measures

Inference Autopsy focuses on externally visible inference behavior:

Metric	Meaning
TTFT	Time from request start to first generated token
TTFB	Time from request start to first response byte
ITL	Inter-token latency between generated output tokens
Request latency	Time from request start to final token or response end
Output TPS	Generated output tokens per second
Stream stalls	Token gaps above a configurable threshold
Error rate	Failed requests divided by total requests
Timeout rate	Timed-out requests divided by total requests
Tail ratio	p99 latency divided by p50 latency

Percentiles are first-class:

p50, p90, p95, p99

Because median latency is where demos look good. Tail latency is where systems start telling the truth.

Target CLI

Run a benchmark

autopsy bench \
  --base-url http://localhost:8000/v1 \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --profile rag-long \
  --concurrency 1,4,8,16 \
  --max-requests 200 \
  --output runs/rag_long_v1.jsonl

Generate a report

autopsy report \
  runs/rag_long_v1.jsonl \
  --html reports/rag_long_v1.html

Diagnose a trace file

autopsy diagnose runs/rag_long_v1.jsonl

Compare two runs

autopsy diff \
  runs/baseline.jsonl \
  runs/candidate.jsonl

Fail CI on regression

autopsy diff \
  runs/baseline.jsonl \
  runs/candidate.jsonl \
  --fail-if "ttft_p95 > +20%" \
  --fail-if "itl_p95 > +15%" \
  --fail-if "error_rate > 1%"

Gate examples:

ttft_p95 > +20%        # relative regression from baseline
latency_p95 > +500ms   # absolute latency increase
error_rate > 1%        # absolute candidate ceiling
error_rate > +2pp      # percentage-point increase
tail_ratio > 3x        # absolute ratio ceiling
output_tps_p50 < -15%  # relative throughput drop

Exit codes:

0  all gates passed
1  valid comparison, but one or more gates failed
2  invalid gate, unreadable trace, or malformed input

Replay a captured workload

autopsy replay \
  runs/baseline.jsonl \
  --base-url http://localhost:11434/v1 \
  --model qwen3:8b \
  --mode shape \
  --output runs/replay_ollama.jsonl

Replay is privacy-aware:

shape replay  regenerates comparable prompts from saved workload metadata
exact replay  requires recoverable full prompts and is refused for hash-only traces

Example Output

Regression detected.

Metric                 Baseline     Candidate    Change
TTFT p95              840ms        1210ms       +44.0%
ITL p95               39ms         47ms         +20.5%
Request p99           6.2s         9.8s         +58.1%
Error rate            0.1%         1.8%         +1.7pp
Output TPS            41.2         35.7         -13.3%

Failed gates:
- ttft_p95 > +20%
- error_rate > 1%

Cause of death:
Tail Wizard + Rate Limit MegaKnight

Failure Arena

Each bad run gets a memorable diagnosis backed by hard metrics.

Cause of death	Serious meaning
TTFT Pekka	First-token latency dominates request time
Decode Barbarian	Inter-token latency is high
Tail Wizard	p99 latency explodes while median looks fine
Context Golem	Long prompts crush prefill performance
Stream Wall Breaker	Streaming has large token gaps
Rate Limit MegaKnight	429s, throttling, or retries dominate
Output Electro Dragon	Output length inflated unexpectedly
JSON Skeleton Army	Structured output mode causes failures or latency
Retry Witch	Hidden retries inflate latency
Queue Queen	Endpoint collapses under parallel load

Example diagnosis:

Cause of death: Context Golem
Severity: High

Evidence:
- TTFT p95 rises from 620ms at 512-token prompts to 3120ms at 8192-token prompts.
- ITL stays mostly flat.
- Request latency increase is concentrated before the first token.

Likely driver:
Long prompt prefill dominates latency.

Suggested next tests:
- Bucket prompts by input length.
- Compare with prefix caching if the backend supports it.
- Test prompt compression.

Workload Profiles

Inference Autopsy uses workload profiles instead of one toy prompt. Different workloads expose different bottlenecks.

Planned built-in profiles:

Profile	Purpose
short-chat	Basic latency and endpoint overhead
rag-long	Long-context RAG-style TTFT sensitivity
code-completion	Long decode and output throughput
agent-json	JSON reliability and structured-output latency
long-context	Context-window and prefill stress
mixed-realistic	Blended production-like workload

Example profile shape:

name: rag-long
description: Long-context RAG-style prompts with moderate outputs.

input_tokens:
  distribution: bucket
  values: [2000, 4000, 8000]
  weights: [0.4, 0.4, 0.2]

output_tokens:
  distribution: normal
  mean: 256
  std: 64
  min: 64
  max: 512

sampling:
  temperature: 0.2
  max_tokens: 512

messages:
  system: "You answer questions using the provided context."
  user_template: |
    Context:
    {{ generated_context }}

    Question:
    {{ generated_question }}

Trace Format

Each request is saved as one JSONL line.

{
  "schema_version": "0.1",
  "run_id": "run_2026_05_23_001",
  "request_id": "req_00042",
  "profile": "rag-long",
  "model": "llama-3.1-8b",
  "base_url_hash": "endpoint_a",
  "input_tokens_estimated": 4096,
  "output_tokens": 261,
  "status": "success",
  "timings_ms": {
    "request_start": 0,
    "first_byte": 817,
    "first_token": 942,
    "request_end": 7610
  },
  "token_times_ms": [942, 971, 1001, 1033, 1208],
  "metrics": {
    "ttft_ms": 942,
    "request_latency_ms": 7610,
    "itl_mean_ms": 28.4,
    "itl_p95_ms": 71.2,
    "output_tps": 38.6,
    "stall_count": 2
  },
  "error": null
}

JSONL is append-friendly, easy to inspect, easy to upload as a CI artifact, and simple to process with Python, DuckDB, Polars, or shell tools.

HTML Reports

The static report is the main demo artifact.

Implemented first-pass sections:

• Executive summary
• Diagnosis cards with evidence
• Summary metric cards
• Overall percentile table
• Static charts
• Profile breakdown
• Concurrency breakdown
• Cache summary
• Worst requests by latency and TTFT
• Methodology notes

The report is designed to be shareable without running a server.

CI Usage

Planned GitHub Actions workflow:

name: LLM Inference Regression Test

on:
  pull_request:

jobs:
  inference-autopsy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install Inference Autopsy
        run: pip install inference-autopsy

      - name: Run benchmark
        run: |
          autopsy bench \
            --base-url ${{ secrets.LLM_BASE_URL }} \
            --api-key ${{ secrets.LLM_API_KEY }} \
            --model ${{ vars.LLM_MODEL }} \
            --profile short-chat \
            --max-requests 50 \
            --output candidate.jsonl

      - name: Check regression
        run: |
          autopsy diff baseline.jsonl candidate.jsonl \
            --fail-if "ttft_p95 > +20%" \
            --fail-if "itl_p95 > +15%" \
            --fail-if "error_rate > 1%"

Compatibility Goal

Inference Autopsy targets OpenAI-compatible chat completion endpoints, especially:

• vLLM OpenAI-compatible server
• Ollama OpenAI-compatible API
• LiteLLM proxy
• hosted OpenAI-compatible inference providers
• internal company deployments using OpenAI-style APIs

Focus;

• /v1/chat/completions
• stream=true
• stream=false
• request-level JSONL traces
• exact replay from saved prompts

Architecture

Typer CLI
  -> async workload runner
  -> OpenAI-compatible HTTP client
  -> streaming parser
  -> JSONL trace recorder
  -> metrics engine
  -> diagnosis engine
  -> report generator
  -> diff and CI gate engine
  -> replay engine

Planned Python stack:

• Typer for the CLI
• httpx for async HTTP
• Pydantic for schemas
• orjson for fast JSON
• Rich for terminal output
• Polars or plain Python for metrics
• Standard-library HTML rendering for the first static report
• Optional Jinja2 and Plotly later when template or chart complexity justifies it
• pytest for tests

Limitations

Inference Autopsy is a black-box endpoint profiler. It identifies externally visible symptoms and likely bottlenecks, not definitive backend internals.

It does not directly observe GPU kernel time, scheduler state, KV-cache pressure, batching internals, or prefill/decode implementation details unless a backend exposes those signals.

Other known limitations:

• Token counting may be approximate when providers do not return usage metadata.
• OpenAI-compatible streaming formats vary across servers.
• Hosted endpoint measurements include network and provider-side variance.
• Replay preserves workload shape and prompts, but not perfect model determinism.
• Static reports are not a replacement for production observability.

Benchmark Methodology

Reports should include:

• endpoint and model
• hardware or provider
• concurrency
• request count
• warmup policy
• timeout policy
• streaming mode
• token counting method
• retry policy
• prompt generation method
• percentile calculation method

No trace, no reproducibility.

Development

python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev]"
pytest
ruff check .
mypy autopsy

Release

Public releases are designed to run through GitHub Actions:

1. Push changes to main.
2. Confirm the CI workflow passes.
3. Confirm the Deploy GitHub Pages workflow publishes the docs site.
4. Create a GitHub release such as v0.1.0.
5. The Publish to PyPI workflow builds and publishes the package through PyPI Trusted Publishing.

The PyPI Trusted Publisher must be configured once in PyPI with:

Repository owner: kaseyho
Repository name: Inference-Autopsy
Workflow name: publish-pypi.yml
Environment name: pypi

License

MIT License. See LICENSE.