localbenchmark 0.1.0

---

Key Features

Feature	Description
Coding benchmark	HumanEval-style tasks via LM Studio; temperature 0 for comparable runs
Long context	Test fixed windows (e.g. 50k–180k); optional context fill so the KV cache is exercised
Speed probe	Prefill / decode tok/s at controlled tokens-in-cache levels
Resume-safe CSVs	Durable results; skip finished configs; retry LLM errors
Slow-model abort	Stop a config after a streak of low decode speed; ranking accounts for incomplete runs
Rank-sum plots	Pass rate, decode speed, TTFT, GPU peak, context — weighted ranking (model × context and per model)
LLMlight helpers	Lightweight local prompting utilities (optional local GGUF via llama-cpp)

---

Requirements

• Python ≥ 3.10
• LM Studio running locally with a model loaded (default http://localhost:1234)
• Dependencies: numpy, pandas, matplotlib, requests, tqdm, openai, scikit-learn, pymupdf, json_repair, datazets

---

Installation

Install from PyPI

pip install localbenchmark

Install from GitHub

pip install git+https://github.com/erdogant/localbenchmark

Install from source

git clone https://github.com/erdogant/localbenchmark
cd localbenchmark
pip install -r requirements.txt
pip install -e .

Check import

import localbenchmark
print(localbenchmark.__version__)

---

Quick start — coding benchmark

Start LM Studio, enable the local server, load a coding model, then:

# Full default suite (models × context windows)
python -m localbenchmark.llm_coding_benchmark

# Subset of models / contexts
python -m localbenchmark.llm_coding_benchmark \
  --models openai/gpt-oss-20b \
  --contexts 50000 80000 100000 \
  --base-url http://localhost:1234

# Stress the window: fill 50% of context with unrelated code before each task
python -m localbenchmark.llm_coding_benchmark --fill 0.5

# Speed probe only (no HumanEval)
python -m localbenchmark.llm_coding_benchmark --probe

# Resume / plots only
python -m localbenchmark.llm_coding_benchmark --plots-only

Useful flags

Flag	Meaning
--models	One or more LM Studio model keys
--contexts	Context sizes in tokens (default includes 50k, 80k, 100k, 120k, 150k, 180k)
--fill	Fraction of the window to fill with filler code (0–0.9)
--limit	Run only the first N HumanEval problems
--results	Results CSV path (default coding_benchmark_results.csv)
--probe	Run the speed probe instead of HumanEval
--no-plots / --plots-only	Skip plots, or only regenerate plots

Results are written to CSV. Configurations already present in the summary CSV are skipped on the next run. Configurations that abort for low tok/s stay in the summary so they are not reloaded endlessly.

---

Plots and ranking

python -m localbenchmark.plot_results \
  --results coding_benchmark_results.csv \
  --out plots \
  --rank-agg sum

# Only large windows (context_actual > 100000)
python -m localbenchmark.plot_results --min-context 100000 --out plots

Flag	Meaning
--rank-agg	Combine per-metric ranks: sum (default), mean, or median
--min-context	Keep rows with context_actual strictly greater than this value

Ranking metrics (lower combined score is better):

1. Pass rate (early aborts count unfinished work against the model)
2. Decode speed (percentage of best — large speed gaps stay large)
3. Time to first token
4. Peak GPU memory
5. Context size (percentage of best)

Pass rate and decode speed are weighted ×2. Figures include pass rate vs context, tok/s, TTFT, GPU memory, quality–speed trade-off, and rank-sum panels (model × context and per model).

---

What is measured

Signal	Why it matters
Pass rate	Generated code must pass the unit tests
Decode speed (tok/s)	Inference has to be usable, not only correct
TTFT	Latency before the first useful token
Peak GPU memory	Whether the setup fits your hardware
Context	Behaviour as the window grows

Default context grid:

50k → 80k → 100k → 120k → 150k → 180k

Any context sizes present in the results file are included in the rank plots (not limited to a hard-coded list).

---

Resources and Links

• Documentation: Website
• Blog: Medium / docs
• Issues: GitHub Issues
• Source: github.com/erdogant/localbenchmark

---

Maintainer

• Erdogan Taskesen, GitHub: erdogant
• Contributions are welcome.
• Yes! This library is entirely free but it runs on coffee! Feel free to support with a Coffee.