🔬 LLMBench

LLM evaluation harness built from scratch.

LLMBench runs structured, reproducible evaluations across four metric families — lexical, semantic, LLM-as-judge, and calibration — stores every run, and diffs any two runs to surface regressions. Plug in any task type or metric in under 10 lines.

Why LLMBench?

Most teams swap models or update prompts and then eyeball outputs to decide if things got better. LLMBench gives you:

• Four metric families — not just ROUGE, but semantic similarity, LLM-as-judge scoring, and calibration (ECE)
• Regression tracking — diff any two run IDs, get a structured report of what regressed
• Full run history — every run stored with config, scores, and per-sample outputs
• Plugin architecture — register custom tasks and metrics in 10 lines
• Any provider — OpenAI, Groq, Anthropic, vLLM, Ollama via one unified interface
• FastAPI + CLI + Dashboard — use it however you want

Quickstart

pip install -e .

# Set your provider API key
export GROQ_API_KEY=your_key_here

# Run an eval
llmbench run tasks/sample_qa.json \
    --task open_qa \
    --model groq/llama-3.3-70b-versatile \
    --metrics exact_match,f1,rouge_l

# List all stored runs
llmbench list

# Compare two runs (regression check)
llmbench compare <baseline_run_id> <candidate_run_id>

# Launch dashboard
streamlit run dashboard/app.py

Python API

from llmbench import build_runner, loader, ModelConfig

# Load a dataset (JSON, CSV, HF Hub, or callable)
dataset = loader.load("squad", task_type="open_qa", max_samples=100)

# Build a runner
runner = build_runner(
    ModelConfig(provider="groq", model_id="llama-3.3-70b-versatile"),
    judge_config=ModelConfig(provider="groq", model_id="llama-3.3-70b-versatile"),
)

# Run the eval
result = runner.run(dataset, metrics=["exact_match", "f1", "bertscore", "llm_relevance"])
print(result.aggregate_scores)

# Save and compare
from llmbench.store.db import store, tracker
store.save(result)

# Later: compare to a previous run
report = tracker.compare(baseline_run_id, result.run_id, threshold=0.02)
print(report["regressions"])   # metrics that dropped > 2%

Metric Reference

Register a custom metric

from llmbench.core.registry import registry

def my_length_metric(results, **_):
    avg_len = sum(len(r.generated_output.split()) for r in results) / len(results)
    return {"avg_output_length": avg_len}

registry.register_metric(
    "avg_output_length",
    "Average word count of generated outputs",
    fn=my_length_metric,
    requires_expected=False,
)

Supported Datasets

Field aliases are resolved automatically: question/query/prompt → input, answer/label/target → expected_output.

Supported Providers

CLI Reference

llmbench run      --task --model [--judge] [--metrics] [--max-samples] [--tag key=val]
llmbench list     [--dataset] [--model] [--task] [--limit]
llmbench compare  <baseline_id> <candidate_id> [--threshold]
llmbench show     <run_id> [--samples] [--top]
llmbench metrics  List all registered metrics
llmbench tasks    List all registered task types

Project Structure

llmbench/
├── llmbench/
│   ├── core/
│   │   ├── schema.py       # EvalSample, EvalDataset, RunResult, ModelConfig
│   │   ├── registry.py     # Task + metric plugin registry
│   │   └── runner.py       # Async batch eval runner
│   ├── loaders/            # JSON, CSV, HF Hub, callable loaders
│   ├── providers/          # OpenAI, Groq, vLLM provider abstraction
│   ├── metrics/            # Lexical, semantic, LLM-judge, calibration
│   ├── store/              # SQLAlchemy results store + regression tracker
│   └── api/                # FastAPI REST + Typer CLI
├── dashboard/              # Streamlit dashboard
├── tasks/                  # Built-in task YAML configs + sample datasets
└── tests/                  # Pytest unit tests

Running Tests

pip install pytest
pytest tests/ -v

All tests run without API keys — providers are mocked at the metric layer.

Roadmap

• Confidence score extraction from log-probs (for ECE on open-source models)
• HTML report export (Jinja2 template)
• GitHub Actions workflow for CI eval gating
• Multi-turn conversation eval support
• Async FastAPI REST endpoint