clean-evals
Measure AI quality across models. Build a golden dataset by working with real data. Pick the right model with the math in plain view.
clean-evals is an open-source application for measuring AI quality
across models. It runs as its own process — a CLI plus a local web UI — and owns its
own data. You bring the prompts and inputs your application sends to a model;
clean-evals measures which model answers them best and what that costs.
Anthropic, OpenAI, Google, and OpenRouter models are supported out of the
box, local models through any OpenAI-compatible server (Ollama, LM Studio,
llama.cpp, vLLM), and anything else through an adapter plugin.
clean-evals is a local tool. It has no authentication and is designed to run on your own machine, bound to localhost. Do not deploy it to a public host or expose its ports to a network you do not fully trust. See Disclaimers.
Capabilities:
- Dataset Builder. Upload your inputs, generate outputs from candidate models, pick or edit the best output, and lock it as the expected answer. Locked cases form the golden dataset.
- Eval Runner. Async, queue-backed (Celery + Redis) or inline, deterministic where the provider allows it, and plugin-extensible, with strict typing and machine- and human-readable output.
- Decision UI. A local web app showing leaderboards, per-case heatmaps, and three model recommendations side by side — maximum accuracy, best price/performance, and lowest cost — with the underlying math visible, plus cost projections.
- Telemetry. Feed real production interactions in — the request, the response, and what the user did next (edited fields, asked for changes, regenerated, accepted). Each interaction derives into a pre-rated candidate case for the golden dataset, and the same signals feed per-model quality monitoring over time.
How it works
- Bring your inputs. The upload wizard asks how your application talks to AI — complete requests, or one system prompt with varying data — and builds the dataset around your answer.
- Generate candidates. Your cases run through the models you select (or a suggested slate of two cheap, two medium, two expensive), producing candidate outputs to compare.
- Review blind and lock golden answers. You rate outputs 1 to 5 without seeing which model wrote them, add feedback, and lock the best answer per case as the reference.
- Calibrate the judge. For open-ended tasks, an LLM judge learns your standard from the ratings and feedback, and its agreement with you is measured before you rely on it.
- Run evals and decide. Runs score the models against the golden dataset and end in a leaderboard with three recommendations — maximum accuracy, best price/performance, lowest cost — with the math in view.
Results drill down from the leaderboard to a per-case heatmap and to the failing cases themselves, with the expected and actual answers side by side:
Telemetry: production data into the golden path
Your users already review model outputs every day — they edit fields, ask for something shorter, hit regenerate, or accept. Telemetry ingestion turns those interactions into dataset material and monitoring signal:
- Two kinds of envelope.
structuredrecords a request, a response, and the user's field edits or accept — derivation is deterministic and free (edit ratio → an implicit 1–5 rating, diffs → feedback, the committed output → the proposed golden answer).transcriptrecords a whole conversation; it is split into per-turn exchanges where the next user message is treated as the review of the previous response, labelled by one small-model classifier call per transcript. - Two ways in. A token-gated batch endpoint
(
POST /api/v1/telemetry/interactions— dark untilCLEAN_EVALS_INGEST_TOKENis set) or a JSONL upload in the UI. Working samples live inexamples/telemetry/. - A human confirms. Derived exchanges land in a review inbox; promotion creates a case, candidate outputs (including discarded regenerations), and an implicit rating. An opt-in auto-lock lane promotes without review only when an explicit accept, the implicit rating, and a calibrated judge all concur — and it routes a sample to spot-checks and disables itself when the human overturn rate climbs.
- Conversations replay faithfully. Promoted transcript cases carry
their prior turns; eval runs send them verbatim as message history
(
request_shape: "chat"), not flattened into a text blob. - Monitoring. Acceptance rate, mean implicit rating, corrections per turn, and turns-to-accept, per source and model over time.
Telemetry is production data: envelopes are stored raw unless you
configure a TelemetryScrubber, and the ingest token protects the ingest
route only — see the
Telemetry guide and
Disclaimers.
Getting started
Manual install (default)
Uses SQLite and local files. No database server or queue is required.
# 1. Install
pip install clean-evals
# 2. Add at least one provider API key to .env
cp .env.example .env
# 3. Initialize the database and start the app
clean-evals migrate
clean-evals serve # http://localhost:8080
Three sample datasets are seeded on first start — ticket triage, sentiment, and summarisation — so you can explore the workflow immediately, including both exact-match scoring and the LLM judge.
Scheduled runs additionally require Redis plus the clean-evals worker and
clean-evals beat processes; docker-compose up starts that full stack
locally.
Docker
git clone https://github.com/datathere/clean-evals
cd clean-evals
cp .env.example .env # fill in API keys
docker-compose up
# UI: http://localhost:8080
All ports published by the compose file bind to 127.0.0.1, so the stack is
reachable only from the machine running it.
Run from source
git clone https://github.com/datathere/clean-evals
cd clean-evals
python -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -e ".[dev]"
# Build the frontend into the package
cd web && npm install && npm run build && cd ..
cp .env.example .env # add API keys
clean-evals migrate
clean-evals serve
Start the app with clean-evals serve, not with uvicorn directly: serve
loads .env, seeds the sample dataset, and warns on non-local binds. Run it
from the directory that contains .env, and restart it after changing keys.
Disclaimers
Read these before relying on clean-evals for anything that matters.
- Not for deployment. There is no authentication or authorization at any
layer. Anyone who can reach the port can read your datasets, edit them, and
trigger runs that spend your provider credits. Run it on localhost only.
The shipped Docker configuration binds every port to
127.0.0.1for this reason. - Cost limits are best-effort, not guarantees. The per-run ceiling
(
--max-cost) is checked as results arrive; calls already in flight complete, so actual spend can overshoot the ceiling. The daily limit (CLEAN_EVALS_DAILY_COST_LIMIT_USD) counts only persisted runs — web and scheduled runs, and CLI runs with--persist. Always verify actual spend in your model provider's billing console. - Cost figures are estimates. Costs are computed from a bundled pricing snapshot (plus any local overrides you configure). Providers change prices; the numbers in reports and recommendations are estimates, not invoices.
- Your data is sent to model providers. Every case input you run is sent to the third-party APIs of the models you select, subject to those providers' data-handling terms.
- PII is not scrubbed automatically. The optional
Scrubberhook applies only to datasets loaded from YAML, and the optionalTelemetryScrubberonly when you configure one. Anything uploaded through the web UI — and any telemetry envelope ingested without a scrubber — is stored as-is. You are responsible for scrubbing your own data. - The telemetry ingest token protects one route. Setting
CLEAN_EVALS_INGEST_TOKENauthenticatesPOST /api/v1/telemetry/interactionsand nothing else. Making that route reachable by your production application does not make the rest of the instance safe to expose; forward only that path through your proxy. - Data is stored in plain text on disk. Prompts, model outputs, datasets,
and rendered reports live unencrypted under
./clean-evals-data/(and your configured database). Treat that directory as sensitive, and note there is no built-in retention or cleanup yet. - The SQLite default is single-user. It is the right zero-configuration choice for one person on one machine. For a shared team instance, use the MySQL or Postgres backends — and put the whole thing on infrastructure you control, never the public internet.
Why clean-evals
- Runs end in a decision. Every eval produces three recommendations side by side — maximum accuracy, best price/performance, lowest cost — with the comparison math in plain view.
- Dataset construction is the first-class flow. Building a golden dataset is most of the real work of evaluation, and the tool is designed around it.
- Everything is inspectable. Readable source, plugin extension points, and dated model snapshots, so every result traces back to the exact model that produced it.
- Boring, typed Python.
mypy --strict, noAny, no magic. The source is meant to be read and trusted quickly.
How ratings and feedback are used
- Ratings and feedback are stored with the golden dataset. In blind review you score a candidate output from 1 to 5 and can add written feedback. Both are saved on that output, under its case in the dataset — they are part of your evaluation data, not throwaway UI state.
- Calibration turns your reviews into the judge's standard. The judge prompt embeds up to six of your rated examples; outputs with written feedback are preferred, and the selection spans the rating range. The judge then scores new outputs against what you demonstrated rather than against a generic rubric.
- Agreement is measured before you trust the judge. During calibration the judge re-scores your rated outputs, drawing its examples from other cases than the one being judged (leave-one-case-out), so the number is not circular. Agreement is reported as the share of outputs scored identically, the share within one point, and Cohen's kappa — 0.6 is the commonly used bar.
- The calibrated rubric becomes the dataset's scorer configuration. Once calibration completes, eval runs are scored by the standard you signed off on, not by a hand-written prompt.
- The judge never sees model names. Review is blind, and the judge is shown inputs, outputs, ratings, and feedback only. It learns your quality bar rather than a brand preference, so it can fairly score models nobody rated during review.
Plugin extension points
| Extension point | Entry-point group | Use case |
|---|---|---|
| Scorer | clean_evals.scorers |
Custom accuracy metrics |
| Adapter | clean_evals.adapters |
Private/internal model gateways |
| Reporter | clean_evals.reporters |
Custom output destinations |
Register via pyproject.toml:
[project.entry-points."clean_evals.scorers"]
my_scorer = "my_package.scorers:MyScorer"
Working with production data
PII handling is your responsibility (see Disclaimers). For
datasets loaded from YAML, a Scrubber plugin you write can clean cases
before they are stored:
from clean_evals import Dataset, Scrubber
class MyScrubber(Scrubber):
def scrub(self, case): ...
ds = Dataset.from_yaml("path.yml", scrubber=MyScrubber())
Documentation
Full documentation at https://datathere.github.io/clean-evals (or
browse the sources below; build locally with mkdocs serve):
- Getting Started
- The Golden Path (product flow)
- Public API Reference
- CLI Reference
- Writing a Scorer
- Writing an Adapter
- Writing a Reporter
- Dataset Builder
- Telemetry
- Running clean-evals
- Brand Policy
License
Copyright (c) 2026 datathere.
clean-evals is open source under the GNU AGPL-3.0. You may use, modify, and redistribute it under the license terms, including commercially. The AGPL's copyleft applies: if you modify clean-evals and make it available to others, including over a network, you must make your modified source code available under the same license.
Commercial licenses are available for organizations that cannot accept
the AGPL's obligations. Contact licenses@datathere.com.
The "clean-evals" and "by datathere" marks are governed by the Brand Use Policy; the in-product attribution is a protected legal notice under section 7(b) of the license and must remain intact.
clean-evals · by datathere · https://github.com/datathere/clean-evals
Metadata
Release files for clean-evals 0.3.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| clean_evals-0.3.0.tar.gz | 551.6 kB | With provenance | Details for clean_evals-0.3.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| clean_evals-0.3.0-py3-none-any.whl | Python 3 | none | any | Details for clean_evals-0.3.0-py3-none-any.whl |
Total release size: 1.1 MB
Release files / clean_evals-0.3.0.tar.gz
| Download URL | clean_evals-0.3.0.tar.gz |
|---|---|
| Size | 551.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
1a7af71f4a00a12c298f40f8734e934ca931a86628e192cdbd2eb754b667c803
|
|
BLAKE2b-256 checksum How to use checksums |
aa7a20670b216f5dd85d7d2835f22937ea194965bb59d5e33f94a101fde17fd9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Aug 27, 2026.
Transparency logRelease files / clean_evals-0.3.0-py3-none-any.whl
| Download URL | clean_evals-0.3.0-py3-none-any.whl |
|---|---|
| Size | 584.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
732c6a53258a313c02cbdeaeca0d08472db1d51359712a91dc3015769fc85401
|
|
BLAKE2b-256 checksum How to use checksums |
9e8cac6396e9bfcc3f614142be83898150569bdb8079fe77c6c31a15fd1ccafe
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Aug 27, 2026.
Transparency logSecurity for clean-evals 0.3.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for clean_evals-0.3.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Aug 27, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.