frugal-llm

How fast, and how expensive, is it to serve your model on your hardware, and what do you lose by serving a cheaper version of it?

frugal-llm points at any OpenAI-compatible endpoint — vLLM, TGI, SGLang, llama.cpp, Ollama or a commercial API — and reports:

• bench: latency percentiles, throughput and cost per million output tokens at each concurrency level you ask for;
• quality and compare: how much a quantised model's answers change against full precision, on shipped task sets, with honest statistics;
• cost: the traffic above which self-hosting beats a commercial API's per-token price.

It speaks only the OpenAI-compatible /v1 API, so there is nothing engine-specific to install or configure.

Quickstart

Requires uv. Against a local Ollama:

ollama pull qwen2.5:0.5b
uvx frugal-llm bench --url http://localhost:11434/v1 --model qwen2.5:0.5b

With the defaults (50 measured requests × 3 repeats, up to 128 output tokens) that takes about 12 minutes on a laptop CPU. For a first look in about 2 minutes:

uvx frugal-llm bench --url http://localhost:11434/v1 --model qwen2.5:0.5b -n 10 --max-tokens 64

A concurrency sweep with a cost model, against vLLM on a GPU you rent for £0.52/hour:

uvx frugal-llm bench --url http://localhost:8000/v1 --model mistral-7b-awq \
  -c 1,4,16,64 -n 200 --cost-per-hour 0.52 --label vllm-awq-L4

Results land in ./results/ as JSON (every individual sample) and Markdown (the table below, ready to paste into an issue or a chat thread).

Example output

The quickstart above, run unmodified on a laptop (Intel Core 7 150U, 16 GB RAM, no GPU, on mains power) against Ollama 0.40.4. This is the Markdown file it wrote, as is:

## frugal-llm — qwen2.5:0.5b @ http://localhost:11434/v1

Config: max_tokens=128, prompt_tokens≈200, requests=50, warmup=5, repeats=3, label=ollama-quickstart

| Conc | TTFT p50 | TTFT p95 | ITL p50 |      Thrpt tok/s | Err |
|-----:|---------:|---------:|--------:|-----------------:|----:|
|    1 |   2222ms |   2408ms |    27ms | 18.1 (18.0-18.4) |  0% |

- Latency percentiles pool the 3 repeats at each level (never across levels). Throughput and cost show the median run (min-max).
- Median output length was below --max-tokens 128 (c=1: 78): the model often stopped early. Throughput and cost reflect these shorter generations.
- Prompt length as counted by the server: median 233 tokens (requested ≈200, including the chat template).
- HTTP/1.1, one connection per concurrent request. Measures speed and cost only; output quality is not measured.

Generated by frugal-llm 0.1.0 on 2026-10-14T17:35:23Z

Read it as: about 2.2 s before the first token appears (prefill of ~233 prompt tokens on a laptop CPU), then a token every ~27 ms, which works out at ~18 tokens/s including that wait. No cost column because no --cost-per-hour was given.

Same model, two servers

The point of measuring rather than guessing. The same model file (Qwen2.5-0.5B-Instruct, Q4_K_M) on the same laptop, swept across concurrency with -c 1,2,4 -n 20 --max-tokens 64:

Ollama 0.40.4 (default settings)

llama.cpp llama-server b11604 (-np 4, four parallel slots)

Ollama, with its defaults on this machine, handles one request at a time: throughput stays flat at ~15 tok/s and every extra concurrent user waits in a queue, so median TTFT grows from 2.2 s to 11.8 s. llama.cpp with four slots processes requests together: median TTFT stays ~1.6 s and throughput rises to ~20 tok/s, paid for with slower token streaming (ITL 30 → 135 ms) as the CPU is shared. Which is better depends on whether your users feel the wait for the first token or the reading speed — which is exactly the trade-off this tool exists to show. These are laptop numbers for a tiny model; they say nothing about your GPU.

Options

Re-aggregate a saved run without re-running it (for example after an upgrade changes how reports are rendered):

uvx frugal-llm report results/20261014T120000Z-vllm-awq-L4.json

What the numbers mean

Latency is only ever reported as percentiles, never a mean: the tail is the point. Each concurrency level is its own distribution; samples are never pooled across levels. Within a level, latency samples from all repeats are pooled, while throughput and cost show the median run with the min–max range across repeats.

Measuring quality loss

New in 0.2.0, and so far exercised only on a small (0.5B) model on development subsets. The result format is provisional; see Results format.

Speed is only half the question. Before serving a cheaper (quantised) model, you want to know what it costs in output quality. frugal-llm answers that relative to the full-precision (FP16) model, on the same hardware and engine:

# FP16 twice (the second run measures run-to-run noise), then the cheaper format
frugal-llm quality --url http://localhost:8080/v1 --model my-model --task mmlu --label fp16-a
frugal-llm quality --url http://localhost:8080/v1 --model my-model --task mmlu --label fp16-b
frugal-llm quality --url http://localhost:8080/v1 --model my-model --task mmlu --label q4_k_m
frugal-llm compare results/*-fp16-a.json results/*-q4_k_m.json --noise-floor results/*-fp16-b.json

• Task sets ship with the tool, so everyone runs identical items: all 1,319 GSM8K test questions (arithmetic reasoning) and a fixed 1,000-question MMLU subset (knowledge, all 57 subjects). Both are MIT-licensed; see src/frugal_llm/tasks/.
• Greedy decoding, fixed prompts, fixed answer-extraction rules, written down before any code existed. Development runs on real models exposed problems (answers read from the wrong place, outputs cut off too early), so some rules were revised before any result was published; every change, and the evidence for it, is logged in docs/quality-method.md.
• compare reports both what changed and whether it matters: accuracy with confidence intervals, the paired difference against FP16 with an exact significance test, how often the final answers agree, and the FP16-versus-FP16 noise floor. It also states the smallest difference the comparison could have detected, so "no difference detected" is never dressed up as "no quality loss".
• Long runs survive interruption: every answer is saved as it arrives, and --resume continues where a run stopped. Rate-limited requests (HTTP 429) are retried, honouring Retry-After, because a quality score needs every question answered.

Is self-hosting cheaper than an API?

New in 0.2.0. The result format is provisional; see Results format.

A rented GPU costs the same per hour busy or idle, so "N× cheaper" claims are really claims about an assumed load. frugal-llm cost answers the real question instead: above what sustained traffic does self-hosting beat paying per token? It works from saved bench results, so prices can be changed without re-running anything:

frugal-llm cost results/<bench-run>.json \
  --hourly-rate 0.44 --rate-source "provider, GPU model, on-demand" --rate-date 2026-10-15 \
  --api "name=some-model,in=0.15,out=0.60,source=https://provider.example/pricing,date=2026-10-15"

For each concurrency level it reports the measured capacity, the cost per 1,000 requests at full use, and against each API price the break-even traffic (requests per day) and break-even utilisation: the share of capacity you must keep busy, around the clock, for self-hosting to cost no more. If even 100% use costs more than the API, it says so. Requests are compared whole, at the measured tokens in and out, because a GPU-hour has no separate input and output price. Rates and prices must carry a source and a date, and are never built into the tool. Method and caveats: docs/cost-method.md.

What this measures and what it doesn't

bench measures speed and cost of generation as seen by a client: how long until the first token, how fast tokens stream, how many tokens per second the endpoint sustains at a given concurrency, and what that costs at the hourly rate you supply.

quality measures how much a cheaper format changes a model's answers on two checkable task sets, relative to FP16 (see above).

Neither measures:

• General output quality. bench measures no quality at all: a fast, cheap model that answers badly will look good there. quality covers arithmetic reasoning and recalled knowledge only, not instruction following, long context, safety or writing. Both task sets are public and may be in a model's training data, so absolute scores are not evidence of general ability; only the comparison against FP16 is meaningful.
• Your real traffic. Load is closed-loop (N requests always in flight), with fixed output length and synthetic or supplied prompts. Bursty traffic, long contexts or variable output lengths will behave differently.
• Network you don't benchmark through. Numbers include the path between the client and the server. Run the client next to the server to measure the server.
• Exact token counts when the server hides them. Output tokens come from the server's usage report; if it doesn't send one, frugal-llm counts content chunks, which undercounts servers that pack several tokens per chunk. The report says which method was used.
• More than the machine's state at the time. Power mode, temperature and other load all move the numbers. Repeats show the spread within one session, not between sessions; benchmark on mains power with nothing else running.
• Cost beyond the hourly rate. Idle time, egress, storage and engineering time are not included.

Why the numbers can be trusted

• Every rule is tested against a mock server that emits chunks at precisely controlled times, so the suite proves the tool measures the server and not itself: TTFT against a known prefill delay, genuinely concurrent requests (past 100), warm-up excluded, levels never pooled, percentile and cost arithmetic by hand.
• Warm-up does double duty. Warm-up requests warm the server and open every worker's connection, so measured requests don't pay TCP/TLS handshakes and the TTFT tail isn't handshake noise.
• Every concurrent worker owns exactly one connection. A shared connection pool either caps you at httpx's default 100 connections (silently serialising the load) or, sized up, burns client CPU matching requests to connections — at 150 concurrent requests that turned a 1 s server delay into a 6 s round.
• The client checks itself. It samples its own event-loop lag during every run and warns in the report when the client, not the server, may have shaped the latencies (for example on an overloaded laptop at high concurrency).
• Prompts are never reused, within a run or across runs (each run gets a random seed, recorded in the results). Servers keep prompt caches between requests; re-sending last run's prompts measured a 28 ms TTFT where a new prompt took ~1000 ms. The server's own cache report is recorded too, and the report warns when cached prompts are flattering TTFT (for example from a repeating --prompts file).
• A machine that sleeps mid-run is caught. The tool keeps the OS awake while it runs, and any run in which the client froze for over a second (a laptop entering standby, say) is excluded from the table and named in the report.
• Failures fail loudly. A preflight request catches a wrong URL, model or key before anything is measured; an unreachable endpoint aborts without writing partial results. Timeouts are per phase and named (read_timeout, total_timeout, ...), so a stalled server is distinguishable from a slow one.
• HTTP/1.1 only. HTTP/2 multiplexing would change what "concurrency" means. Recorded in every result file.

Results format

Every run writes JSON with every individual sample, so results can be re-analysed later (report, compare and cost all work from saved files). Each file records its schema_version (0.1 for bench, quality-0.1 for quality) and the full environment: the client machine, what the server reported about itself, and your --meta key=value notes.

The format is provisional ("schema_status": "provisional" in every file) until the first full study on real hardware is complete. Quality and cost measurement have so far run only against a small model, and the real study may show that something must be recorded differently. Until then, results from different versions may not be directly comparable, and there is no process yet for contributing results; both will come once the format is stable.

Development

uv sync
uv run pytest
uv run ruff check . && uv run ruff format --check .

The tests need no GPU and no model: they run against tests/mock_server.py, which you can also start by hand (uv run python tests/mock_server.py) and point the CLI at.

License

Apache-2.0. See LICENSE.