Skip to main content

clean-evals

Measure AI quality across models. Build a golden dataset by working with real data. Pick the right model with the math in plain view.

PyPI version Python versions License: AGPL-3.0 CI

clean-evals is an open-source application for measuring AI quality across models. It runs as its own process — a CLI plus a local web UI — and owns its own data. You bring the prompts and inputs your application sends to a model; clean-evals measures which model answers them best and what that costs. Anthropic, OpenAI, Google, and OpenRouter models are supported out of the box, local models through any OpenAI-compatible server (Ollama, LM Studio, llama.cpp, vLLM), and anything else through an adapter plugin.

clean-evals is a local tool. It has no authentication and is designed to run on your own machine, bound to localhost. Do not deploy it to a public host or expose its ports to a network you do not fully trust. See Disclaimers.

Capabilities:

  1. Dataset Builder. Upload your inputs, generate outputs from candidate models, pick or edit the best output, and lock it as the expected answer. Locked cases form the golden dataset.
  2. Eval Runner. Async, queue-backed (Celery + Redis) or inline, deterministic where the provider allows it, and plugin-extensible, with strict typing and machine- and human-readable output.
  3. Decision UI. A local web app showing leaderboards, per-case heatmaps, and three model recommendations side by side — maximum accuracy, best price/performance, and lowest cost — with the underlying math visible, plus cost projections.
  4. Telemetry. Feed real production interactions in — the request, the response, and what the user did next (edited fields, asked for changes, regenerated, accepted). Each interaction derives into a pre-rated candidate case for the golden dataset, and the same signals feed per-model quality monitoring over time.

A completed eval run: leaderboard and model recommendations

How it works

  1. Bring your inputs. The upload wizard asks how your application talks to AI — complete requests, or one system prompt with varying data — and builds the dataset around your answer.
  2. Generate candidates. Your cases run through the models you select (or a suggested slate of two cheap, two medium, two expensive), producing candidate outputs to compare.
  3. Review blind and lock golden answers. You rate outputs 1 to 5 without seeing which model wrote them, add feedback, and lock the best answer per case as the reference.
  4. Calibrate the judge. For open-ended tasks, an LLM judge learns your standard from the ratings and feedback, and its agreement with you is measured before you rely on it.
  5. Run evals and decide. Runs score the models against the golden dataset and end in a leaderboard with three recommendations — maximum accuracy, best price/performance, lowest cost — with the math in view.

Results drill down from the leaderboard to a per-case heatmap and to the failing cases themselves, with the expected and actual answers side by side:

Eval results: per-case heatmap and failed cases with expected and actual answers

Telemetry: production data into the golden path

Your users already review model outputs every day — they edit fields, ask for something shorter, hit regenerate, or accept. Telemetry ingestion turns those interactions into dataset material and monitoring signal:

  • Two kinds of envelope. structured records a request, a response, and the user's field edits or accept — derivation is deterministic and free (edit ratio → an implicit 1–5 rating, diffs → feedback, the committed output → the proposed golden answer). transcript records a whole conversation; it is split into per-turn exchanges where the next user message is treated as the review of the previous response, labelled by one small-model classifier call per transcript.
  • Two ways in. A token-gated batch endpoint (POST /api/v1/telemetry/interactions — dark until CLEAN_EVALS_INGEST_TOKEN is set) or a JSONL upload in the UI. Working samples live in examples/telemetry/.
  • A human confirms. Derived exchanges land in a review inbox; promotion creates a case, candidate outputs (including discarded regenerations), and an implicit rating. An opt-in auto-lock lane promotes without review only when an explicit accept, the implicit rating, and a calibrated judge all concur — and it routes a sample to spot-checks and disables itself when the human overturn rate climbs.
  • Conversations replay faithfully. Promoted transcript cases carry their prior turns; eval runs send them verbatim as message history (request_shape: "chat"), not flattened into a text blob.
  • Monitoring. Acceptance rate, mean implicit rating, corrections per turn, and turns-to-accept, per source and model over time.

Telemetry monitoring: per-source acceptance and per-model quality over time

Telemetry is production data: envelopes are stored raw unless you configure a TelemetryScrubber, and the ingest token protects the ingest route only — see the Telemetry guide and Disclaimers.

Getting started

Manual install (default)

Uses SQLite and local files. No database server or queue is required.

# 1. Install
pip install clean-evals

# 2. Add at least one provider API key to .env
cp .env.example .env

# 3. Initialize the database and start the app
clean-evals migrate
clean-evals serve          # http://localhost:8080

Three sample datasets are seeded on first start — ticket triage, sentiment, and summarisation — so you can explore the workflow immediately, including both exact-match scoring and the LLM judge.

Scheduled runs additionally require Redis plus the clean-evals worker and clean-evals beat processes; docker-compose up starts that full stack locally.

Docker

git clone https://github.com/datathere/clean-evals
cd clean-evals
cp .env.example .env       # fill in API keys
docker-compose up
# UI: http://localhost:8080

All ports published by the compose file bind to 127.0.0.1, so the stack is reachable only from the machine running it.

Run from source

git clone https://github.com/datathere/clean-evals
cd clean-evals
python -m venv .venv
source .venv/bin/activate            # Windows: .venv\Scripts\activate
pip install -e ".[dev]"

# Build the frontend into the package
cd web && npm install && npm run build && cd ..

cp .env.example .env                 # add API keys
clean-evals migrate
clean-evals serve

Start the app with clean-evals serve, not with uvicorn directly: serve loads .env, seeds the sample dataset, and warns on non-local binds. Run it from the directory that contains .env, and restart it after changing keys.

Disclaimers

Read these before relying on clean-evals for anything that matters.

  • Not for deployment. There is no authentication or authorization at any layer. Anyone who can reach the port can read your datasets, edit them, and trigger runs that spend your provider credits. Run it on localhost only. The shipped Docker configuration binds every port to 127.0.0.1 for this reason.
  • Cost limits are best-effort, not guarantees. The per-run ceiling (--max-cost) is checked as results arrive; calls already in flight complete, so actual spend can overshoot the ceiling. The daily limit (CLEAN_EVALS_DAILY_COST_LIMIT_USD) counts only persisted runs — web and scheduled runs, and CLI runs with --persist. Always verify actual spend in your model provider's billing console.
  • Cost figures are estimates. Costs are computed from a bundled pricing snapshot (plus any local overrides you configure). Providers change prices; the numbers in reports and recommendations are estimates, not invoices.
  • Your data is sent to model providers. Every case input you run is sent to the third-party APIs of the models you select, subject to those providers' data-handling terms.
  • PII is not scrubbed automatically. The optional Scrubber hook applies only to datasets loaded from YAML, and the optional TelemetryScrubber only when you configure one. Anything uploaded through the web UI — and any telemetry envelope ingested without a scrubber — is stored as-is. You are responsible for scrubbing your own data.
  • The telemetry ingest token protects one route. Setting CLEAN_EVALS_INGEST_TOKEN authenticates POST /api/v1/telemetry/interactions and nothing else. Making that route reachable by your production application does not make the rest of the instance safe to expose; forward only that path through your proxy.
  • Data is stored in plain text on disk. Prompts, model outputs, datasets, and rendered reports live unencrypted under ./clean-evals-data/ (and your configured database). Treat that directory as sensitive, and note there is no built-in retention or cleanup yet.
  • The SQLite default is single-user. It is the right zero-configuration choice for one person on one machine. For a shared team instance, use the MySQL or Postgres backends — and put the whole thing on infrastructure you control, never the public internet.

Why clean-evals

  • Runs end in a decision. Every eval produces three recommendations side by side — maximum accuracy, best price/performance, lowest cost — with the comparison math in plain view.
  • Dataset construction is the first-class flow. Building a golden dataset is most of the real work of evaluation, and the tool is designed around it.
  • Everything is inspectable. Readable source, plugin extension points, and dated model snapshots, so every result traces back to the exact model that produced it.
  • Boring, typed Python. mypy --strict, no Any, no magic. The source is meant to be read and trusted quickly.

How ratings and feedback are used

  • Ratings and feedback are stored with the golden dataset. In blind review you score a candidate output from 1 to 5 and can add written feedback. Both are saved on that output, under its case in the dataset — they are part of your evaluation data, not throwaway UI state.
  • Calibration turns your reviews into the judge's standard. The judge prompt embeds up to six of your rated examples; outputs with written feedback are preferred, and the selection spans the rating range. The judge then scores new outputs against what you demonstrated rather than against a generic rubric.
  • Agreement is measured before you trust the judge. During calibration the judge re-scores your rated outputs, drawing its examples from other cases than the one being judged (leave-one-case-out), so the number is not circular. Agreement is reported as the share of outputs scored identically, the share within one point, and Cohen's kappa — 0.6 is the commonly used bar.
  • The calibrated rubric becomes the dataset's scorer configuration. Once calibration completes, eval runs are scored by the standard you signed off on, not by a hand-written prompt.
  • The judge never sees model names. Review is blind, and the judge is shown inputs, outputs, ratings, and feedback only. It learns your quality bar rather than a brand preference, so it can fairly score models nobody rated during review.

Plugin extension points

Extension point Entry-point group Use case
Scorer clean_evals.scorers Custom accuracy metrics
Adapter clean_evals.adapters Private/internal model gateways
Reporter clean_evals.reporters Custom output destinations

Register via pyproject.toml:

[project.entry-points."clean_evals.scorers"]
my_scorer = "my_package.scorers:MyScorer"

Working with production data

PII handling is your responsibility (see Disclaimers). For datasets loaded from YAML, a Scrubber plugin you write can clean cases before they are stored:

from clean_evals import Dataset, Scrubber

class MyScrubber(Scrubber):
    def scrub(self, case): ...

ds = Dataset.from_yaml("path.yml", scrubber=MyScrubber())

Documentation

Full documentation at https://datathere.github.io/clean-evals (or browse the sources below; build locally with mkdocs serve):

License

Copyright (c) 2026 datathere.

clean-evals is open source under the GNU AGPL-3.0. You may use, modify, and redistribute it under the license terms, including commercially. The AGPL's copyleft applies: if you modify clean-evals and make it available to others, including over a network, you must make your modified source code available under the same license.

Commercial licenses are available for organizations that cannot accept the AGPL's obligations. Contact licenses@datathere.com.

The "clean-evals" and "by datathere" marks are governed by the Brand Use Policy; the in-product attribution is a protected legal notice under section 7(b) of the license and must remain intact.


clean-evals · by datathere · https://github.com/datathere/clean-evals

Metadata

Release files for clean-evals 0.3.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for clean-evals 0.3.0
File Size Uploaded Provenance Actions
clean_evals-0.3.0.tar.gz 551.6 kB With provenance Details for clean_evals-0.3.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for clean-evals 0.3.0
File Interpreter ABI Platform Actions
clean_evals-0.3.0-py3-none-any.whl Python 3 none any Details for clean_evals-0.3.0-py3-none-any.whl

Total release size: 1.1 MB

Release files / clean_evals-0.3.0.tar.gz

Download URL clean_evals-0.3.0.tar.gz
Size 551.6 kB
Tags Source
SHA-256 checksum
How to use checksums
1a7af71f4a00a12c298f40f8734e934ca931a86628e192cdbd2eb754b667c803
BLAKE2b-256 checksum
How to use checksums
aa7a20670b216f5dd85d7d2835f22937ea194965bb59d5e33f94a101fde17fd9
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Aug 27, 2026.

Transparency log

Release files / clean_evals-0.3.0-py3-none-any.whl

Download URL clean_evals-0.3.0-py3-none-any.whl
Size 584.2 kB
Tags Python 3
SHA-256 checksum
How to use checksums
732c6a53258a313c02cbdeaeca0d08472db1d51359712a91dc3015769fc85401
BLAKE2b-256 checksum
How to use checksums
9e8cac6396e9bfcc3f614142be83898150569bdb8079fe77c6c31a15fd1ccafe
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Aug 27, 2026.

Transparency log

Security for clean-evals 0.3.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for clean_evals-0.3.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Aug 27, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page