🏎️ bench-my-llm

New here? Start with the Getting Started Guide.

Stop guessing which model is faster. Measure it.

Point bench-my-llm at any OpenAI-compatible API and get latency, throughput, cost, and quality metrics in seconds. Compare models side by side. Get a beautiful terminal report. Ship with confidence.

✨ Features

• 🔥 TTFT Measurement - Time to first token via streaming
• ⚡ Tokens per Second - Real throughput numbers
• 📊 p50 / p95 / p99 Latencies - Production-grade percentiles
• 💰 Cost Estimation - Know what you're spending
• 🎯 Quality Scoring - Compare responses against reference answers
• 🏁 Model Comparison - Side-by-side with winner highlights
• 📦 Built-in Prompt Suites - Reasoning, coding, creative, factual
• 🔌 Any OpenAI-compatible API - OpenAI, Anthropic, Ollama, vLLM, Together, and more
• 💾 Export to JSON, CSV, Markdown, HTML - Pipe into CI, dashboards, or share a report
• 📈 Historical Trends - Track quality, speed, and cost per model over time with sparklines
• 🦙 Ollama Auto-Detection - Find local models and benchmark them with a single flag
• 🧮 Custom Prompt Suites - Load your own domain-specific prompts from JSON or YAML files
• 📦 Suite Packs - Install, update, and publish shareable prompt suites by name from a simple JSON registry
• 🧾 Run Metadata and Notes - Every saved result records the endpoint, temperature, suite, tool version, and an optional note so old numbers stay interpretable
• 🔬 Metadata Filtering and Diffing - Filter saved runs to like-for-like configs and diff any two to see which metadata changed and how the numbers moved
• 🏷️ Metadata Grouping - Group saved runs by endpoint, temperature, model, or tool version and compare aggregate metrics per group

🚀 Quick Start

pip install bench-my-llm

Single Model Benchmark

bench-my-llm run --model gpt-4o --suite reasoning

┌──────────────────────────────────────────────────────────┐
│  🏎️  Benchmark Report                                    │
│  bench-my-llm results for gpt-4o                         │
│  Suite: reasoning | Prompts: 5 | Cost: $0.0043           │
└──────────────────────────────────────────────────────────┘

          Latency Summary
┌────────┬────────────┬────────────────────┐
│ Metric │ TTFT (ms)  │ Total Latency (ms) │
├────────┼────────────┼────────────────────┤
│ p50    │ 234.1      │ 1,523.4            │
│ p95    │ 312.7      │ 2,187.9            │
│ p99    │ 348.2      │ 2,401.3            │
│ Mean   │ 251.3      │ 1,687.2            │
└────────┴────────────┴────────────────────┘

       Throughput & Quality
┌───────────────────┬─────────────┐
│ Metric            │ Value       │
├───────────────────┼─────────────┤
│ Mean TPS          │ 67.3 tok/s  │
│ Median TPS        │ 64.8 tok/s  │
│ Quality Score     │ 82%         │
│ Estimated Cost    │ $0.0043     │
└───────────────────┴─────────────┘

Model Comparison

bench-my-llm compare gpt-4o gpt-4o-mini --suite reasoning

┌──────────────────────────────────────────────────────────┐
│  🏁 Model Comparison                                     │
│  gpt-4o vs gpt-4o-mini                                   │
└──────────────────────────────────────────────────────────┘

              Head-to-Head
┌────────────────────────┬─────────┬─────────────┐
│ Metric                 │ gpt-4o  │ gpt-4o-mini │
├────────────────────────┼─────────┼─────────────┤
│ TTFT p50 (ms)          │ 234.1   │ 142.3  🏆   │
│ TTFT p95 (ms)          │ 312.7   │ 198.4  🏆   │
│ Total Latency p50 (ms) │ 1523.4  │ 876.2  🏆   │
│ Mean TPS               │ 67.3 🏆 │ 54.1        │
│ Cost (USD)             │ $0.0043 │ $0.0008 🏆  │
│ Quality Score          │ 0.82 🏆 │ 0.71        │
└────────────────────────┴─────────┴─────────────┘

🏆 Winner: gpt-4o-mini (4/6 metrics)

📖 Usage

Custom Prompt Suites

Benchmark with your own domain-specific prompts by passing a JSON or YAML file path to --suite:

bench-my-llm run --model gpt-4o --suite my_suite.json
bench-my-llm compare gpt-4o gpt-4o-mini --suite ./suites/support.yaml

Suite file format (see examples/custom_suite.json for a full example):

{
  "name": "support-bot",
  "description": "Customer support prompts",
  "prompts": [
    {
      "text": "Explain our 30-day return policy in two sentences.",
      "category": "support",
      "reference": "Items can be returned within 30 days for a full refund.",
      "max_tokens": 256
    }
  ]
}

Only text is required per prompt. category defaults to custom, reference to empty (skips quality scoring), and max_tokens to 512. Files are validated with friendly error messages, including typo detection for unknown keys. YAML files need the optional extra: pip install 'bench-my-llm[yaml]'.

Suite Packs

Share suites with your team and install them by name. A registry is a JSON document (local file or HTTP URL) mapping pack names to suite files:

{
  "packs": {
    "support-basics": {
      "version": "1.0.0",
      "description": "Customer support prompts",
      "url": "support_basics.json"
    }
  }
}

Pack URLs can be absolute (http, https, or a filesystem path) or relative to the registry location, so a registry can live next to its suite files in a repo or on any static file host.

# Install a pack (validated before it lands on disk)
bench-my-llm suite install support-basics --registry https://example.com/registry.json

# Or set a default registry once
export BENCH_MY_LLM_REGISTRY=https://example.com/registry.json
bench-my-llm suite install support-basics

# See what is installed, then benchmark with a pack by name
bench-my-llm suite list
bench-my-llm run --model gpt-4o --suite support-basics

# Remove a pack
bench-my-llm suite remove support-basics

Keep installed packs current with suite update. It re-fetches each pack's registry from the source recorded at install time, reinstalls packs whose registry version changed, and reports the rest as up to date.

# Check and update every installed pack
bench-my-llm suite update

# Update specific packs only
bench-my-llm suite update support-basics other-pack

# Reinstall even when the version is unchanged
bench-my-llm suite update --force

# Point at a different registry, or get machine-readable results
bench-my-llm suite update support-basics -r new-registry.json
bench-my-llm suite update --json-output

Update failures (a registry that moved, a pack pulled from its registry) are reported per pack without aborting the rest, and the command exits nonzero if anything failed.

Installed packs live in ~/.bench-my-llm/suites (override the base directory with BENCH_MY_LLM_HOME). Try it locally with the bundled example: bench-my-llm suite install example-basics -r examples/registry.json.

Publishing your own packs is the reverse direction: validate a suite file and add it to a registry in one step.

# Validate a suite and print its registry entry as JSON
bench-my-llm suite publish my_suite.json

# Write the entry into a registry file (created if missing)
bench-my-llm suite publish my_suite.json -r registry.json -n my-pack -v 1.0.0

# Bump an existing pack
bench-my-llm suite publish my_suite.json -r registry.json -n my-pack -v 1.1.0 --force

The suite is validated before anything is written, existing registry entries are preserved, and the URL is stored relative to the registry when the suite file lives in the same directory tree, so the pair stays portable. Host the resulting registry anywhere static files can be served and teammates can suite install from it.

Prompt Suites

Suite	Description	Prompts
reasoning	Logic, math, step-by-step	5
coding	Code generation and explanation	5
creative	Writing, storytelling, metaphors	5
factual	Knowledge recall, definitions	5
all	Everything combined	20

Export Results

bench-my-llm run --model gpt-4o --suite all --output results.json
bench-my-llm report results.json

# Convert saved results to other formats
bench-my-llm export results.json --format markdown
bench-my-llm export results.json --format csv -o results.csv

Run Metadata and Notes

Every benchmark run captures its environment automatically: endpoint, temperature, suite (name and prompt count), tool version, and a UTC timestamp. Add a free-form note with --note to remember why you ran it:

bench-my-llm run --model gpt-4o --suite reasoning \
  --note "baseline before prompt rewrite" --output baseline.json

The terminal report prints a Run Metadata panel, and the saved JSON gains a metadata object:

{
  "metadata": {
    "notes": "baseline before prompt rewrite",
    "endpoint": "https://api.openai.com/v1",
    "temperature": 0.0,
    "suite_version": "reasoning (5 prompts)",
    "tool_version": "0.3.0",
    "captured_at": "2026-08-29T05:00:00+00:00"
  }
}

Older result files without a metadata block still load fine. Python API: RunMetadata and the metadata field on BenchmarkRun.

HTML Reports

Turn any saved results file into a shareable, self-contained HTML report with summary cards, latency tables, per-prompt quality bars, and a model comparison section for multi-model files. No JavaScript, no external assets, safe to attach to a PR or email:

bench-my-llm export results.json --format html -o report.html
bench-my-llm export comparison.json --format html --title "GPT vs Claude" -o report.html
open report.html

Cost-Adjusted Leaderboard

Rank saved runs by value, not just raw speed. The composite score weighs quality (50%), cost (30%, lower is better), and throughput (20%):

bench-my-llm leaderboard results/*.json
bench-my-llm leaderboard a.json b.json --sort quality-per-dollar
bench-my-llm leaderboard results/*.json --json-output

Sort options: value (default), quality, cost, speed, quality-per-dollar.

Historical Trends

Save each benchmark run to a results directory, then track how every model's numbers move over time. Runs are grouped per model and ordered chronologically, with a terminal sparkline and the change from the first run to the latest:

bench-my-llm run -m gpt-4o -s reasoning -o results/gpt4o-$(date +%F).json
bench-my-llm trends ./results/
bench-my-llm trends ./results/ --metric latency
bench-my-llm trends ./results/ -m gpt-4o --metric cost --json-output

Model     Runs  Trend      First   Latest  Change
gpt-4o       5  ▂▃▅▆█        61%      74%  ▲ +21.3%
llama3       4  ▅▄▃▂         55%      48%  ▼ -12.7%

Metrics: quality (default), tps, ttft, latency, cost. For ttft, latency, and cost, lower is better and the arrows account for that.

Filter and Diff Runs by Metadata

Every saved run carries a metadata block (endpoint, temperature, tool version, suite). Use it to narrow a results directory down to like-for-like configurations, then diff any two runs to see which metadata changed and how the numbers moved:

bench-my-llm runs filter ./results/ --endpoint api.openai.com -t 0
bench-my-llm runs filter ./results/ --tool-version 0.4 --json-output
bench-my-llm runs diff ./results/ before.json after.json

runs filter matches --model, --endpoint, and --tool-version on substrings and --temperature exactly; with no criteria it lists every run. runs diff selects each run by file name or a unique substring and prints a metadata change table plus per-metric deltas (using the same lower-is-better arrows as trends), so an environment change that shifted the results is easy to spot:

        Metadata Changes
Field         Old    New
temperature   0      0.7
tool_version  0.3.0  0.4.0

              Metric Changes
Metric   Old      New      Change
latency  800ms    400ms    ▲ -50.0%

Group Runs by Metadata

To compare whole configurations instead of run pairs, group a results directory by a metadata field and read aggregate metrics per group:

bench-my-llm runs group ./results/ --by endpoint
bench-my-llm runs group ./results/ --by temperature --model gpt-4o
bench-my-llm runs group ./results/ --by tool_version --json-output

Grouping fields are model (the default), suite, endpoint, temperature, tool_version, and suite_version. Each group reports the mean quality, tokens per second, TTFT p50, latency p50, and cost across its member runs; --json-output adds min, max, and the member file names. Runs without a value for the field land in an (unset) group. This answers questions like "how did average quality differ between my two endpoints across all runs" in one command:

   Runs Grouped by endpoint (5 run(s), 2 group(s))
endpoint                   Runs  Quality  TTFT p50   Cost
http://localhost:1234/v1      3      72%     220ms  $0.0000
https://api.openai.com/v1     2      89%      95ms  $0.0041

Local Models (Ollama)

bench-my-llm auto-detects a running Ollama server. List what is installed locally:

bench-my-llm ollama

              Local Ollama Models (2)
Model         Params  Quant    Size    Family
llama3:8b     8B      Q4_0     4.7 GB  llama
phi3:mini     3.8B    Q4_K_M   2.3 GB  phi3

Then benchmark with the --ollama flag, no URL or key needed. Tagless names resolve automatically (llama3 finds llama3:latest):

bench-my-llm run --ollama -m llama3
bench-my-llm compare --ollama llama3 phi3 -s coding
bench-my-llm ollama --json-output          # machine-readable model list
bench-my-llm ollama --url http://gpu:11434 # remote Ollama host

If a model is not installed, the error lists everything that is. Manual configuration still works for any OpenAI-compatible endpoint:

bench-my-llm run --model llama3 --base-url http://localhost:11434/v1 --api-key ollama

CI Integration

Add to your GitHub Actions workflow:

- name: Benchmark LLM
  run: |
    pip install bench-my-llm
    bench-my-llm run --model gpt-4o-mini --suite reasoning --output benchmark.json
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

- name: Upload results
  uses: actions/upload-artifact@v4
  with:
    name: benchmark-results
    path: benchmark.json

🛠️ Development

git clone https://github.com/manasvardhan/bench-my-llm.git
cd bench-my-llm
python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
pytest

📄 License

MIT. See LICENSE.