frugal-llm

How fast, and how expensive, is it to serve your model on your hardware?

frugal-llm points at any OpenAI-compatible endpoint — vLLM, TGI, SGLang, llama.cpp, Ollama or a commercial API — and reports latency percentiles, throughput and cost per million output tokens at each concurrency level you ask for. It speaks only POST /v1/chat/completions with streaming, so there is nothing engine-specific to install or configure.

Quickstart

Requires uv. Against a local Ollama:

ollama pull qwen2.5:0.5b
uvx frugal-llm bench --url http://localhost:11434/v1 --model qwen2.5:0.5b

With the defaults (50 measured requests × 3 repeats, up to 128 output tokens) that takes about 12 minutes on a laptop CPU. For a first look in about 2 minutes:

uvx frugal-llm bench --url http://localhost:11434/v1 --model qwen2.5:0.5b -n 10 --max-tokens 64

A concurrency sweep with a cost model, against vLLM on a GPU you rent for £0.52/hour:

uvx frugal-llm bench --url http://localhost:8000/v1 --model mistral-7b-awq \
  -c 1,4,16,64 -n 200 --cost-per-hour 0.52 --label vllm-awq-L4

Results land in ./results/ as JSON (every individual sample) and Markdown (the table below, ready to paste into an issue or a chat thread).

Example output

The quickstart above, run unmodified on a laptop (Intel Core 7 150U, 16 GB RAM, no GPU, on mains power) against Ollama 0.40.4. This is the Markdown file it wrote, as is:

## frugal-llm — qwen2.5:0.5b @ http://localhost:11434/v1

Config: max_tokens=128, prompt_tokens≈200, requests=50, warmup=5, repeats=3, label=ollama-quickstart

| Conc | TTFT p50 | TTFT p95 | ITL p50 |      Thrpt tok/s | Err |
|-----:|---------:|---------:|--------:|-----------------:|----:|
|    1 |   2222ms |   2408ms |    27ms | 18.1 (18.0-18.4) |  0% |

- Latency percentiles pool the 3 repeats at each level (never across levels). Throughput and cost show the median run (min-max).
- Median output length was below --max-tokens 128 (c=1: 78): the model often stopped early. Throughput and cost reflect these shorter generations.
- Prompt length as counted by the server: median 233 tokens (requested ≈200, including the chat template).
- HTTP/1.1, one connection per concurrent request. Measures speed and cost only; output quality is not measured.

Generated by frugal-llm 0.1.0 on 2026-10-14T17:35:23Z

Read it as: about 2.2 s before the first token appears (prefill of ~233 prompt tokens on a laptop CPU), then a token every ~27 ms, which works out at ~18 tokens/s including that wait. No cost column because no --cost-per-hour was given.

Same model, two servers

The point of measuring rather than guessing. The same model file (Qwen2.5-0.5B-Instruct, Q4_K_M) on the same laptop, swept across concurrency with -c 1,2,4 -n 20 --max-tokens 64:

Ollama 0.40.4 (default settings)

llama.cpp llama-server b11604 (-np 4, four parallel slots)

Ollama, with its defaults on this machine, handles one request at a time: throughput stays flat at ~15 tok/s and every extra concurrent user waits in a queue, so median TTFT grows from 2.2 s to 11.8 s. llama.cpp with four slots processes requests together: median TTFT stays ~1.6 s and throughput rises to ~20 tok/s, paid for with slower token streaming (ITL 30 → 135 ms) as the CPU is shared. Which is better depends on whether your users feel the wait for the first token or the reading speed — which is exactly the trade-off this tool exists to show. These are laptop numbers for a tiny model; they say nothing about your GPU.

Options

Re-aggregate a saved run without re-running it (for example after an upgrade changes how reports are rendered):

uvx frugal-llm report results/20261014T120000Z-vllm-awq-L4.json

What the numbers mean

Latency is only ever reported as percentiles, never a mean: the tail is the point. Each concurrency level is its own distribution; samples are never pooled across levels. Within a level, latency samples from all repeats are pooled, while throughput and cost show the median run with the min–max range across repeats.

What this measures and what it doesn't

It measures speed and cost of generation as seen by a client: how long until the first token, how fast tokens stream, how many tokens per second the endpoint sustains at a given concurrency, and what that costs at the hourly rate you supply.

It does not measure:

• Output quality or accuracy. Not at all, in v0.1. A fast, cheap model that answers badly will look good here. Evaluate quality separately before deciding.
• Your real traffic. Load is closed-loop (N requests always in flight), with fixed output length and synthetic or supplied prompts. Bursty traffic, long contexts or variable output lengths will behave differently.
• Network you don't benchmark through. Numbers include the path between the client and the server. Run the client next to the server to measure the server.
• Exact token counts when the server hides them. Output tokens come from the server's usage report; if it doesn't send one, frugal-llm counts content chunks, which undercounts servers that pack several tokens per chunk. The report says which method was used.
• More than the machine's state at the time. Power mode, temperature and other load all move the numbers. Repeats show the spread within one session, not between sessions; benchmark on mains power with nothing else running.
• Cost beyond the hourly rate. Idle time, egress, storage and engineering time are not included.

Why the numbers can be trusted

• Every rule is tested against a mock server that emits chunks at precisely controlled times, so the suite proves the tool measures the server and not itself: TTFT against a known prefill delay, genuinely concurrent requests (past 100), warm-up excluded, levels never pooled, percentile and cost arithmetic by hand.
• Warm-up does double duty. Warm-up requests warm the server and open every worker's connection, so measured requests don't pay TCP/TLS handshakes and the TTFT tail isn't handshake noise.
• Every concurrent worker owns exactly one connection. A shared connection pool either caps you at httpx's default 100 connections (silently serialising the load) or, sized up, burns client CPU matching requests to connections — at 150 concurrent requests that turned a 1 s server delay into a 6 s round.
• The client checks itself. It samples its own event-loop lag during every run and warns in the report when the client, not the server, may have shaped the latencies (for example on an overloaded laptop at high concurrency).
• Prompts are never reused, within a run or across runs (each run gets a random seed, recorded in the results). Servers keep prompt caches between requests; re-sending last run's prompts measured a 28 ms TTFT where a new prompt took ~1000 ms. The server's own cache report is recorded too, and the report warns when cached prompts are flattering TTFT (for example from a repeating --prompts file).
• A machine that sleeps mid-run is caught. The tool keeps the OS awake while it runs, and any run in which the client froze for over a second (a laptop entering standby, say) is excluded from the table and named in the report.
• Failures fail loudly. A preflight request catches a wrong URL, model or key before anything is measured; an unreachable endpoint aborts without writing partial results. Timeouts are per phase and named (read_timeout, total_timeout, ...), so a stalled server is distinguishable from a slow one.
• HTTP/1.1 only. HTTP/2 multiplexing would change what "concurrency" means. Recorded in every result file.

Development

uv sync
uv run pytest
uv run ruff check . && uv run ruff format --check .

The tests need no GPU and no model: they run against tests/mock_server.py, which you can also start by hand (uv run python tests/mock_server.py) and point the CLI at.

License

Apache-2.0. See LICENSE.