llm-grill

CLI for benchmarking LLM inference servers: vLLM, SGLang, llama.cpp, LiteLLM.

Measures TTFT, TPOT, end-to-end latency, throughput, success rate, KV cache quality metrics, and load ramp (breaking-point detection) on multi-turn conversation scenarios.

Install

Requires Python 3.11+ and uv.

uv tool install llm-grill

Verify:

llm-grill --version

Quick start

Copy the example scenario and adapt it to your setup:

cp scenarios/example.yaml scenarios/my-bench.yaml
# Edit URLs, model name, and API key

1. Check connectivity

llm-grill ping scenarios/my-bench.yaml

2. Run a benchmark

llm-grill run scenarios/my-bench.yaml --output results.jsonl

After the run, tables are printed automatically:

• Benchmark Summary — latency, throughput, success rate per server/model
• Conversation Quality Metrics — KV cache hit rate, turn-to-turn latency ratio, context growth factor
• Load Ramp Results — (if ramp_levels is set) one row per (server, model, concurrency level)

3. Generate a report from an existing results file

# Terminal table (summary + conversation metrics)
llm-grill report results.jsonl

# JSON (both sections, pipeable)
llm-grill report results.jsonl --format json

# CSV (raw requests, pandas-ready)
llm-grill report results.jsonl --format csv --output summary.csv

# Hide conversation metrics table
llm-grill report results.jsonl --no-conversations

Commands

run options

report options

Global options

Supported backends

Scenario format (YAML)

name: my-scenario
description: Optional description

backends:
  - name: gpu-vllm
    url: http://gpu-vllm:8000
    api_key: none                    # "none", a literal key, or ${ENV_VAR}
    type: vllm                       # vllm | sglang | llamacpp | litellm | openai
    timeout: 120.0

models:
  - name: devstral-small-2-24b
    max_tokens: 512
    temperature: 0.0

conversations:
  - name: multi-turn-debug
    turns:
      - role: system
        content: "You are an expert developer."
      - role: user
        content: "My FastAPI app returns 500 errors under load. What should I check?"
      - role: user
        content: "The DB connection pool is exhausted. How do I configure it in SQLAlchemy?"

targets:
  - backend: gpu-vllm
    model: devstral-small-2-24b
    conversation: multi-turn-debug

load:
  concurrent_users: 10
  iterations: 3
  ramp_up_seconds: 5.0
  think_time_seconds: 0.0

Each role: user turn triggers an inference request. Conversation history (including assistant responses) is carried forward, so the server sees a growing context.

Load ramp

Add ramp_levels to sweep concurrency levels in a single run. When set, concurrent_users is ignored.

load:
  iterations: 3
  ramp_levels: [1, 5, 10, 20, 50, 100]
  ramp_pause_seconds: 10.0   # pause between levels, default 10 s
  think_time_seconds: 0.0

Results are tagged with concurrent_users_level in the JSONL output and displayed in a Load Ramp Results table sorted by (server, model, users).

Metrics

Latency & throughput

t0       → request sent
t_first  → first non-empty content chunk received
t_last   → stream ends ([DONE] or connection close)

TTFT   = t_first - t0
E2E    = t_last  - t0
TPOT   = (E2E - TTFT) / max(completion_tokens - 1, 1)

Measurement includes network round-trip. For cross-server comparisons, run from the same network location.

Conversation quality (multi-turn)

Computed per (server, model, conversation) group:

GPU monitoring

Enable per-backend GPU metrics (utilization, memory, temperature, power) collected via SSH:

backends:
  - name: gpu-vllm
    url: http://gpu-vllm:8000
    type: vllm
    gpu_monitoring: true
    ssh_host: gpu-vllm       # defaults to URL host if omitted
    ssh_user: root            # default

Requires nvidia-smi on the target host and SSH key-based access.

Output format (JSONL)

One JSON object per request, written incrementally:

{
  "scenario": "my-scenario",
  "target_server": "gpu-vllm",
  "target_model": "devstral-small-2-24b",
  "conversation": "multi-turn-debug",
  "turn": 1,
  "iteration": 0,
  "user_id": 3,
  "timestamp_start": "2026-03-10T14:00:00+00:00",
  "ttft_s": 0.142,
  "tpot_s": 0.018,
  "e2e_latency_s": 1.23,
  "prompt_tokens": 45,
  "completion_tokens": 64,
  "tokens_per_second": 52.0,
  "success": true,
  "error": null,
  "kv_cache_usage": 0.34,
  "requests_running": 8.0,
  "concurrent_users_level": 10
}

The file is valid even if the benchmark is interrupted — each line is a complete record.

Read with pandas:

df = pd.read_json("results.jsonl", lines=True)
df.groupby("target_server")[["ttft_s", "e2e_latency_s", "tokens_per_second"]].mean()

Read with polars:

df = pl.read_ndjson("results.jsonl")
df.group_by("target_server").agg(pl.col("ttft_s").mean())

API keys

Use ${ENV_VAR} syntax to read from environment variables at load time:

backends:
  - name: gateway
    url: http://my-litellm-proxy:4000
    api_key: ${LITELLM_API_KEY}
    type: litellm

export LITELLM_API_KEY="sk-..."
llm-grill run scenarios/my-scenario.yaml

Never commit literal API keys in scenario files.

LiteLLM gateway routing

When backends are behind a LiteLLM proxy, define one backend entry for the gateway and use model aliases to route:

backends:
  - name: gateway
    url: http://my-litellm-proxy:4000
    api_key: ${LITELLM_API_KEY}
    type: litellm

models:
  - name: devstral-small-llama    # LiteLLM alias → llama.cpp
    max_tokens: 512
  - name: devstral-small-vllm     # LiteLLM alias → vLLM
    max_tokens: 512

targets:
  - backend: gateway
    model: devstral-small-llama
    conversation: short-code-question
  - backend: gateway
    model: devstral-small-vllm
    conversation: short-code-question

Aliases must match model_name values in LiteLLM's config.yaml.

Troubleshooting

Contributing

See CONTRIBUTING.md.

License

Apache 2.0 — see LICENSE.