Llamathon

Put your LLM server through a marathon.
One command to benchmark its inference capacity — real workloads, race-ready reports.

📊 Voir un rapport réel →  —  live report.html from an actual run, no setup.

llamathon is an orchestrator around llama-benchy. It:

1. Finds the model(s) — either the name(s) you pass, or by auto-detecting what is installed on the machine where the script runs (Ollama, an OpenAI-compatible endpoint, or local .gguf files).
2. Sizes the workload from a real prompt + result — it counts their token length with the model's own tokenizer, so the benchmark reflects a realistic request instead of arbitrary default token counts.
3. Runs llama-benchy against the endpoint and exports JSON stats.
4. Renders charts (PNG) and a Markdown report so results are easy to read, share, and compare between models and servers.

Built to be dropped onto a server and run unattended.

Table of contents

• Example report
• Quick start
• How it works
• Installation
• Usage
• Exit codes
• Model auto-detection
• Token counting
• Output
• Server deployment
• Development
• Troubleshooting
• FAQ

Example report

Real output from a run on a 12-core CPU laptop (no GPU), racing qwen2.5-coder:7b against qwen2.5:0.5b through Ollama.

report.html — one self-contained page: server specs, latency percentiles per concurrency level, token load, model comparison, dashboard and Pareto chart:

dashboard.png — throughput & latency curves vs concurrency, one line per model:

pareto.png — throughput vs latency trade-off, one point per concurrency level:

Quick start

# 1. Install (Python 3.10+)
pip install llamathon

# 2. Make sure your model server is up and OpenAI-compatible.
#    e.g. Ollama exposes http://localhost:11434/v1 automatically.

# 3. Benchmark everything installed, sizing load from a real prompt:
llamathon \
  --base-url http://localhost:11434/v1 \
  --prompt "Summarise the causes of World War I in 5 bullet points." \
  --result prompts/sample_result.txt

# 4. Look in results/run_<timestamp>/ for report.html, REPORT.md,
#    dashboard.png and pareto.png

Benchmark a single named model:

llamathon --model llama3.1:8b --base-url http://localhost:11434/v1

python benchmark.py still works as a backwards-compatible alias for the llamathon command.

How it works

             ┌──────────────┐  detection.py — auto-detect (concurrent probes,
 --model ───▶│   resolve    │  priority: ollama list → GET /v1/models → *.gguf,
             │   models     │  first source with hits wins)
             └──────┬───────┘
                    │  for each model
   --prompt ─┐      ▼
   --result ─┴▶┌──────────────┐  tokenization.py — transformers → tiktoken →
               │ count tokens │  heuristic  => --pp (prompt) / --tg (generation)
               └──────┬───────┘
                      ▼
               ┌──────────────┐  runner.py — uvx / llama-benchy --format json
               │ llama-benchy │  (timeout, retries, stderr captured)
               └──────┬───────┘
                      ▼
               ┌──────────────┐  metrics.py + charts.py + reporting.py
               │charts+reports│  => dashboard.png, pareto.png, REPORT.md,
               └──────────────┘     report.html

The code lives in the llamathon/ package (one module per stage above, plus constants.py for every default/tunable); benchmark.py is a thin compatibility shim and cli.py the entry point.

Installation

Python package

pip install llamathon                # core: llama-benchy + matplotlib
pip install "llamathon[tiktoken]"    # + fast generic token counting (recommended)
pip install "llamathon[tokens]"      # + transformers for exact per-model counts

Token counting degrades gracefully: if neither transformers nor tiktoken is installed, the tool falls back to a char/word heuristic and says so in the logs and report.

From source

git clone https://github.com/qthullie/Llamathon
cd Llamathon
python -m pip install .               # or ".[tiktoken]" / ".[tokens]"

(pip install -r requirements.txt still works and mirrors the core + tiktoken set.)

llama-benchy

Installed as a dependency of this package. If you prefer the zero-install path, install uv instead and the script will invoke uvx llama-benchy automatically — no separate install needed.

Resolution order the script uses to find the engine:

1. llama-benchy on PATH
2. uvx llama-benchy
3. uv tool run llama-benchy

A model server

llama-benchy talks to an OpenAI-compatible endpoint. Any of these work:

Usage

llamathon [options]

Examples

Compare two models under load, exact output length, deep context:

llamathon \
  --model llama3.1:8b --model qwen2.5:7b \
  --base-url http://localhost:8000/v1 \
  --prompt prompts/sample_prompt.txt \
  --result prompts/sample_result.txt \
  --runs 5 --concurrency 1 2 4 --depth 0 4096 --exact-tg

Pass an advanced flag straight through to llama-benchy:

llamathon --model llama3.1:8b --extra --latency-mode generation --no-warmup

Racing several models produces side-by-side curves, a comparison table and a Pareto chart:

Exit codes

Designed so cron/CI can react to partial failures:

Model auto-detection

When --model is omitted the tool probes three sources concurrently, then applies a priority order — the first source that yields models wins:

1. Ollama — ollama list (if the ollama binary is on PATH).
2. Endpoint — GET {base_url}/models on the OpenAI-compatible server (with retries + exponential backoff on transient network failures).
3. Local weights — recursive scan of --gguf-dir for *.gguf files.

Sources are not merged: mixing them would benchmark names the target endpoint can't serve (e.g. local .gguf stems against a remote vLLM). When several sources have hits, the ignored ones are listed in the log — pass --model to override. Duplicates are removed by name. If nothing is found, the tool exits with code 3 and a clear message telling you to pass --model.

Token counting

The point of feeding a real prompt + result: the benchmark's prompt-processing size (--pp) and generation size (--tg) match what the model would actually handle, so throughput/latency numbers are meaningful.

Counting strategy (first available wins):

1. transformers AutoTokenizer — exact, per-model (needs the model name to be a valid HuggingFace repo id, or use --tokenizer).
2. tiktoken cl100k_base — a solid generic BPE approximation.
3. Heuristic — ~4 chars/token, floored at the whitespace word count.

The method actually used is printed and recorded in REPORT.md. Tokenizers are cached per name, so multi-model runs never reload the same tokenizer.

Security note: custom tokenizer code from HuggingFace repos (trust_remote_code) is disabled by default — it would execute arbitrary code from the repo. Enable with --trust-remote-code only for tokenizers you trust.

Output

Each run creates results/run_<YYYYMMDD_HHMMSS>/ containing:

results/run_20260710_143000/
├── stats_llama3.1_8b.json     # raw llama-benchy JSON, one per model
├── stats_qwen2.5_7b.json
├── dashboard.png              # one clean multi-panel figure (see below)
├── pareto.png                 # throughput-vs-latency trade-off (concurrency sweep)
├── report.html               # self-contained report (charts embedded) — open in a browser
└── REPORT.md                  # Markdown tables + failures + chart

See Example report above for what these actually look like.

pareto.png (when sweeping concurrency): aggregate throughput (y, higher better) vs first-token latency p95 (x, log, lower better), one point per concurrency level. The top-left corner is ideal; it shows at a glance which concurrency buys throughput without wrecking latency, and which levels are dominated.

dashboard.png adapts to the data:

• Several concurrency (or depth) levels → line charts — throughput/latency curves vs load, one line per model, with a legend and direct labels.
• A single point → stat tiles — a big number per metric instead of a lonely bar. (This is why the default sweeps --concurrency 1 2 4 8.)

Parsed metrics (llama-benchy ≥0.4.0): prompt throughput (pp_throughput), generation throughput (tg_throughput), peak throughput, time-to-first-response (ttfr), estimated prompt-processing time (est_ppt), end-to-end TTFT. Missing metrics are skipped.

report.html is the one to download — a single self-contained file (opens in any browser, light/dark aware) that adds, on top of the dashboard:

• Detail by concurrency — a per-configuration table with throughput, scaling efficiency, and TTFR p50 / p95 / p99 percentiles (tail latency, not just the mean).
• Server specs — hostname / OS / CPU / RAM / GPU of the machine that ran the inference (numbers are meaningless without the hardware).
• Failures — any model that failed, with the llama-benchy error tail.
• Side-by-side model comparison + token load tables.

Readability choices baked into the charts:

• Latency (time-to-first-response) is shown as the median with a shaded p50–p95 band, not mean ± std — tail latency is what hurts and the mean hides it.
• Total throughput is derived (per-request speed × concurrency) so you can see whether the server actually scales or just queues.
• Y-axis switches to log scale automatically when a metric spans >30×.
• Every panel is tagged ↑ better / ↓ better.

Server deployment

The script is headless-safe (matplotlib Agg backend) and writes only into --out-dir. Typical patterns:

One-shot on a remote box

ssh gpu-server
git clone <this-repo> && cd Llamathon
python -m pip install ".[tiktoken]"
llamathon --base-url http://localhost:11434/v1 \
  --prompt prompts/sample_prompt.txt --result prompts/sample_result.txt

Scripted wrapper

Use run.sh (POSIX) or run.ps1 (Windows) to pin env vars per server:

BENCHY_BASE_URL=http://localhost:8000/v1 ./run.sh

Scheduled (cron) capacity check

# Nightly at 02:30, benchmark all installed models.
# Exit code 5 = partial failure, worth alerting on.
30 2 * * *  cd /opt/Llamathon && llamathon \
             --base-url http://localhost:11434/v1 \
             --prompt prompts/sample_prompt.txt \
             --result prompts/sample_result.txt \
             >> /var/log/llamathon.log 2>&1

Containerised

A Dockerfile ships with the repo (slim Python 3.11 base, tiktoken included):

docker build -t llamathon .
docker run --rm --network host \
  -e BENCHY_BASE_URL=http://localhost:11434/v1 \
  -v "$PWD/results:/app/results" llamathon \
  --prompt prompts/sample_prompt.txt

--network host lets the container reach a model server running on the host.

Development

pip install -e ".[dev,tiktoken]"
ruff check . && ruff format llamathon tests   # lint + format
mypy llamathon                                # type-check
pytest                                           # unit + integration tests

CI (GitHub Actions) runs lint, type-check and the test suite on Python 3.10–3.12 for every push/PR. The integration test mocks both the OpenAI-compatible endpoint and the llama-benchy binary — no model server needed. See CONTRIBUTING.md and CHANGELOG.md.

Troubleshooting

FAQ

Does it need a GPU? No — the script itself is light. The model server it benchmarks may use a GPU; that's what you're measuring.

Does it work fully offline? Yes, if tiktoken is installed (no network) and you pass --model explicitly. transformers may fetch a tokenizer on first use; the tiktoken/heuristic fallbacks are offline.

Can it benchmark a remote endpoint? Yes — point --base-url at any reachable OpenAI-compatible URL. Auto-detection of .gguf/Ollama only sees the local box, but endpoint /models detection works remotely.

Where's the raw data? stats_*.json per model — untouched llama-benchy output, safe to post-process yourself.

License

MIT.