Evalstand

Status: 1.0.2, stable. Install with pip install evalstand.

Why

Evaluating an LLM application should feel like running a test suite.

evalstand is a local-first LLM evaluation tool for Python. You write an eval file, run a watch command, and results stream into a live terminal UI - scores, nested call traces, token counts, latency, and cost. Everything runs on your machine and persists to a local SQLite database, so you can compare a run against the one before it.

Existing Python options are either heavyweight platforms that push you toward a hosted service, or bare metric libraries with no runner, no persistence, and no live feedback loop. evalstand is the middle: a real runner with a real UI that stays on your machine.

How to start

git clone https://github.com/MiltonKlun/Evalstand && cd Evalstand
uv sync

# qa_eval.py
from evalstand import Case, evaluate, llm
from evalstand.scorers import exact


async def answer(question: str) -> str:
    reply = await llm.acall("gpt-4o-mini", [{"role": "user", "content": question}])
    return reply.text.strip()


evaluate(
    name="capitals",
    cases=[
        Case(id="france", input="Capital of France? City only.", expected="Paris"),
        Case(id="japan", input="Capital of Japan? City only.", expected="Tokyo"),
        Case(id="peru", input="Capital of Peru? City only.", expected="Lima"),
    ],
    task=answer,
    scorers=[exact],
)

export OPENAI_API_KEY=sk-...
evalstand watch          # the live view, re-running when you edit
evalstand run            # one pass, prints a summary
evalstand serve          # browse past runs in a browser (needs the web extra)
pytest qa_eval.py        # the plain test runner; same runner underneath

Three things, and only three: Cases are the inputs, the Task is your function under test, and Scorers judge what it returned.

What you get

Docs

• Quickstart — install, write an eval, run it
• Writing evals — cases, tasks, repeats, custom columns
• Scorers — the library, and writing your own
• Traces — what your task did, and what it cost
• Watching — the live view and watch mode
• CI — thresholds, exit codes, pull-request comments
• Web UI — serve, the JSON API, and how to read its numbers
• Architecture — how the pieces fit, for anyone changing them
• Limitations — what it does not do, stated plainly
• Decisions — why the design is the way it is

In CI

evalstand run --threshold 0.85 --fail-on-error

0 met the bar, 1 fell below it, 2 something did not run. --output markdown produces a body for a pull-request comment. See docs/ci.md for the workflow and the full table.

Development

uv sync --all-extras --dev
uv run pytest                       # the suite
uv run python scripts/mutate.py     # 292 mutants, all killed
uv run mkdocs serve                 # the docs site

The mutation harness is the real quality measure here. Every defect this project has found ships with a mutant that reintroduces it, so a test that stops catching its bug fails loudly rather than passing quietly. tests/unit/test_mutation_harness.py holds the harness itself to the same standard — a stale anchor reports as a broken probe, not as a survivor.

Licence

This project is licensed under the MIT License.

Author

Milton Klun
QA Automation Engineer | AI Quality Testing

_{Inspired by evalite (MIT), which showed that local LLM evals could feel like running tests. evalstand is an independent Python implementation.}