llm-eval-bench

A lightweight, local CLI tool for evaluating LLM prompt outputs against YAML-defined test suites.

No account. No cloud. No lock-in. Just pip install and a YAML file.

---

Why

Most LLM evaluation tools are cloud-based, require accounts, and send your data to third-party servers. llm-eval-bench runs entirely on your machine — your prompts and outputs stay local.

---

Features

• Rule-based evaluators — contains, not_contains, max_words, min_words, regex, starts_with
• Semantic similarity — embedding-based similarity scoring using sentence-transformers, runs fully locally
• LLM-as-judge — use any supported model to score outputs against a rubric
• Model comparison — run the same suite against two models and compare results side by side
• Multiple providers — OpenAI, Anthropic, Google Gemini
• JSON output — save results to a file for CI/CD pipelines

---

Install

pip install llm-eval-bench

---

Quick Start

1. Create a test suite:

# my_suite.yaml
suite: "My Prompt Tests"
model: gpt-4o-mini

tests:
  - name: "Capital city"
    prompt: "What is the capital of France?"
    expect:
      - type: contains
        value: "Paris"
      - type: max_words
        value: 50

  - name: "Code generation"
    prompt: "Write a Python function to reverse a string. Return only the code."
    expect:
      - type: contains
        value: "def "
      - type: regex
        value: "def \\w+\\("

  - name: "Tone check"
    prompt: "Write a formal email declining a job offer"
    expect:
      - type: not_contains
        value: "hey"
      - type: llm_judge
        rubric: "Is this email professional and polite?"
        score_threshold: 4
        out_of: 5

2. Run it:

llm-eval run --suite my_suite.yaml --api-key YOUR_API_KEY

3. See results:

llm-eval-bench — My Prompt Tests
...
Results: 3/3 passed  |  Avg latency: 1.4s  |  Avg tokens: 54

Supported Models

Provider	Models
OpenAI	gpt-4o, gpt-4o-mini, gpt-3.5-turbo
Anthropic	claude-sonnet-4-6, claude-haiku-4-5-20251001
Gemini	gemini-2.0-flash, gemini-1.5-flash, gemini-1.5-pro
Hugging Face	meta-llama/Llama-3.1-8B-Instruct, mistralai/Mistral-7B-Instruct-v0.3

---

Evaluator Reference

Rule-based

expect:
  - type: contains
    value: "Paris"

  - type: not_contains
    value: "I cannot"

  - type: max_words
    value: 50

  - type: min_words
    value: 10

  - type: regex
    value: "def \\w+\\("

  - type: starts_with
    value: "The"

Semantic Similarity

Runs locally using sentence-transformers. No API key needed.

expect:
  - type: semantic_similarity
    expected: "Water boils at 100 degrees Celsius"
    threshold: 0.75  # 0.0 to 1.0, default 0.75

LLM-as-Judge

expect:
  - type: llm_judge
    rubric: "Is this explanation clear and accurate?"
    score_threshold: 4   # minimum passing score
    out_of: 5            # maximum score

---

Commands

Run a suite

llm-eval run --suite my_suite.yaml --api-key YOUR_KEY

Options:

• --model — override the model set in the YAML file
• --judge-model — use a different model as the LLM judge
• --output results.json — save results to a JSON file

Compare two models

llm-eval compare \
  --suite my_suite.yaml \
  --baseline gpt-4o \
  --candidate gpt-4o-mini \
  --baseline-api-key YOUR_KEY \
  --candidate-api-key YOUR_KEY

Save results to JSON

llm-eval run --suite my_suite.yaml --api-key YOUR_KEY --output results.json

---

Environment Variables

Instead of passing --api-key every time, set environment variables:

export LLM_API_KEY=your-key-here
export JUDGE_API_KEY=your-judge-key-here  # optional, defaults to LLM_API_KEY

---

Use in CI/CD

# .github/workflows/eval.yml
- name: Run LLM eval
  run: |
    pip install llm-eval-bench
    llm-eval run --suite tests/prompts.yaml --output results.json
  env:
    LLM_API_KEY: ${{ secrets.OPENAI_API_KEY }}

---

Project Structure

llm_eval/
├── cli.py           # entry point
├── runner.py        # orchestrates the full eval pipeline
├── loader.py        # parses YAML/JSON test suites
├── reporter.py      # terminal and JSON output
├── adapters/        # one file per LLM provider
└── evaluators/      # rule_based, semantic, llm_judge

Custom Evaluators

Write your own evaluator without modifying core code:

1. Create your evaluator file:

# my_evaluator.py
from llm_eval.evaluators.base import BaseEvaluator

class NoSlangEvaluator(BaseEvaluator):
    SLANG = ["gonna", "wanna", "gotta", "kinda", "yeah", "nope"]

    def evaluate(self, output: str, **kwargs) -> dict:
        found = [w for w in self.SLANG if w in output.lower().split()]
        passed = len(found) == 0
        return {
            "passed": passed,
            "reason": f"Slang found: {found}" if not passed else "No slang detected",
            "score": None,
        }

EVALUATOR = NoSlangEvaluator()

2. Reference it in your YAML:

expect:
  - type: custom
    custom_evaluator_path: "my_evaluator.py"
    custom_evaluator_name: "no_slang"

License

MIT — see LICENSE