anyeval

Check that an AI evaluation repository conforms to the AnyEval standard, and run it end to end to get real numbers.

pipx run anyeval-cli check .

Why

A benchmark score is comparable to another score only when five things match: the model, the provider that served it, the eval, the problem, and the harness. Published numbers routinely state one and omit four, which makes them impossible to reproduce and unsafe to compare — including for whoever published them.

The AnyEval standard is a repository layout and a manifest that make all five recordable. This tool decides whether a repository conforms, so conformance is an objective result rather than a review opinion.

Commands

anyeval check .                  # 27 rules, pass/fail each, exit 1 on failure
anyeval check . --json           # machine-readable, for CI
anyeval spec                     # print the standard
anyeval run . --sample-id 3 --model qwen/qwen3-32b

check never runs your code

It reads files and parses run.py with ast. It does not import or execute anything in the repository, so it is safe to point at a stranger's checkout. The trade is stated plainly: static analysis can confirm that run.py declares --sample-id and refuses glob patterns, not that it honours them at runtime. That is what run is for.

run does run your code

Running an eval is the only way to prove it works, so anyeval run executes the repository's own run.py against a live model and prints the score, tokens, cost, and which provider actually served the request.

It needs a TrustedRouter key in TRUSTEDROUTER_API_KEY. The run is billed to your account; this tool sends the key to nothing but TrustedRouter.

It refuses to start on a non-conforming repository unless you pass --force, because the rules are what stop one problem becoming a full-dataset bill.

The most expensive rule

runner.rejects_globs exists because Inspect matches sample ids with fnmatch and only warns when a glob selects many samples — it errors solely when the filtered set is empty. So --sample-id '*' silently runs the entire dataset while billing for one problem. Every conforming run.py refuses glob metacharacters outright.

Exit status

0 when the repository conforms, 1 when it does not, 2 on a usage error. Suitable for CI.

Submitting an eval

Check it, run it, then email the repository URL to anyeval-submission@trustedrouter.com. See anyeval.com/submit.

You do not have to submit at all — the standard and this tool are open, and neither depends on anyeval.com. A format only usable by its author is not a standard.

Note on the package name

The anyeval name on PyPI belongs to an unrelated project, so this package is published as anyeval-cli. It installs both an anyeval and an anyeval-cli command.

Licence

Apache-2.0