3tears-evals

Evals for LLM features, from a classifier to a tool-using agent: it runs each version over the same cases and says what moved, with the uncertainty that says whether the move is real. Python 3.14+.

pip install 3tears-evals

Rung 0 is one TOML file and no Python. examples/1-triage/triage.toml triages 30 support emails, each with the triage a person gave it, three times each:

# Rung 0, no Python: python -m threetears.evals triage.toml
cases = "emails.jsonl"
grader = ["category", "urgent", "needs_human"]
recording = "recordings/triage-toml.jsonl"
k = 3

[runner]
model = "deepseek/deepseek-v4-flash"
prompt = "Triage this support email:\n\n{email}"

[runner.output]
category = ["billing", "bug", "account", "sales", "spam"]
urgent = "bool"
needs_human = "bool"

python -m threetears.evals examples/1-triage/triage.toml replays the recorded model, offline and free, and its first line says how it went:

triage: 16/30 cases passed; 14 misses: gdpr-delete, nonprofit, phish-suspend, phish-invoice, crash-at-checkout, and 9 more

Below it come each field's pass rate, each category label's precision and recall, the top misses with why each missed, and where its report page is. Then climb the ladder, one idea per rung:

Where the science lives:

• Measuring soundly: which design answers your question, how many cases, and the traps.
• Judges and calibration: whether to trust a judge, and how far.
• Reading reports: every verdict, interval and disclosure a report prints.

Wiring it into your app (your store, your launcher, your world): Adopting the engine.

Install

pip install 3tears-evals is the base install. The FastMCP transport, for serving evals to an agent, is the [fastmcp] extra:

pip install "3tears-evals[fastmcp]"

To install a pre-release or an unreleased checkout, install from the repository at a tag instead:

pip install "3tears-evals @ git+https://github.com/pacepace/3tears.git@<tag>#subdirectory=packages/evals"

In a checkout of this repo, uv sync at the root does it, and the commands above run from packages/evals with uv run in front. threetears.evals.__version__ says which version is installed. The completion client (Model) is on the base install: it speaks each provider's HTTP API over httpx, with no SDK and no other 3tears package. The examples run on cheap models through OpenRouter (OPENROUTER_API_KEY); Model(id, provider="anthropic") (ANTHROPIC_API_KEY) and provider="openai" (OPENAI_API_KEY) are options, and base_url= takes any OpenAI-compatible endpoint. After a pip install, fetch the examples from the repository first (how).

The public API is still changing. From 0.70.0 only its host contract is promised: a name on it moves with a deprecation cycle, and any other declared name may change in any release, recorded in MIGRATIONS.toml. About 520 names are declared, mapped by rung; the rest a root exports are provisional (stability tiers).

What is different

• A verdict, not a mean. Every version is run over the same frozen cases and tested case by case against a control, and evidence that cannot tell two versions apart says "not separated", never "no difference" (why).
• Guardrails apart from capability. What a version must never do is decided on its own, so a gain cannot pay for a breach (why).
• A judge is measured, not trusted. Every judged reading carries the evidence tier its judge's agreement with people and with itself reached (how).
• An agent is graded on the world it leaves, a world your app declares, with checks proven to beat doing nothing and a conformance kit your app runs against its own world (what is new).

Docs

These links are relative to the package, so on GitHub each opens the page at the ref you are reading; on PyPI, where this README is the package's description, they do not resolve, so read the docs in the repository at your tag.

Climb

• Building an inventory: what to test before a line of code, ending in rung 0; then which cases a classifier needs, and how many.
• The ladder: rungs 1 to 4, through the four examples.
• Concepts: the six concepts, and the glossary of every other word.

Go deeper

• Judges and calibration: rubrics, labels, and whether to trust the judge.
• Reading reports: what each part of a comparison and a report means.
• Measuring soundly: which design answers a question, how many cases, and the traps campaigns have hit.
• Evaluating a tool-using agent: a world, seeds, goal checks, the conformance kit, and why the world contract is shaped as it is.
• Cost and budgets: what a launch will cost, and what stops it.
• Serving: the command line, keeping and reading runs, the actions an agent calls, and MCP.
• Adopting the engine: your own host, store, kind and launcher.

Background and reference

• Design: the principles the engine keeps, why each concept is shaped as it is, and the prior art.
• Open problems: known gaps.
• The surface: the declared names on one page, by rung.
• Reference: every declared name, setting, report field, goal-check builtin and action, generated from the code.

For experts

Import only from the public roots the reference lists: a module below a root is internal and may move. A public name that leaves a root is recorded with its new home in each release's notes (Upgrading: names that move).