beetle-evals 0.1.0

TinyEvals for your AI agents and LLM/SLM apps.
_{Zero runtime dependencies · ~2,100 lines of Python · numbers you can put in a merge decision}

Install · Quick start · How it works · Evaluators · Judges · CLI · Examples · Contributing

Beetle runs your system over a suite of cases, scores each run and tells you whether today is worse than yesterday. Zero runtime dependencies, roughly 2,100 lines of Python.

Most eval harnesses make it easy to produce a number and hard to trust it. Beetle leans the other way. An evaluator that crashes is recorded as an error rather than a zero, no mean is reported without a sample size and an interval next to it, and a run that mostly failed cannot present itself as a run that mostly passed.

Install

pip install git+https://github.com/Neilblaze/beetle

Python 3.10 or newer. Nothing else — the wheel carries no runtime dependency, which CI verifies against the built package's metadata on every push. Linux and macOS are covered in CI.

For a checkout you intend to hack on, see CONTRIBUTING.md.

Quick start

from beetle import Case, run
from beetle.evaluators import Contains

report = run([Case("2+2?", expected="4")], lambda case: "4", [Contains("4")])
print(report.pass_rate)

A task is any callable that takes a Case. An evaluator is any object with evaluate(case, result) -> Score, or a plain function of the same shape, so there is nothing to register and no plugin system to learn. Writing your own takes one function.

from beetle import Score


def mentions_units(case, result):
    ok = "km" in str(result.output)
    return Score.of("mentions_units", 1.0 if ok else 0.0, ok, "found km" if ok else "no units")

Score.of asks you to state passed explicitly rather than deriving it from a threshold, and Score.error(name, reason) is there for the case where you could not judge at all. Reaching for the second one instead of scoring a 0.0 is the whole difference between a report that is honest and one that is merely confident.

render(report) gives you the run as text, which is also what the command line prints.

run 20260816T154204Z-a27bb538  (3 case(s) x 1 trial(s))
beetle 0.1.0 | python 3.12.12 | seed 0 | concurrency 1

cases: 2 passed, 1 failed, 0 errored/timed out  (pass rate  66.7% of judged)

evaluator                      pass     n  err  mean
Contains                      66.7%     3    0   0.667 [0.21, 0.94]

1 case(s) did not pass:
  FAIL     four
           Contains = 0.00  'the answer is' not present in output

How it works

flowchart LR
    LIB("beetle.run"):::e
    CLI("beetle CLI"):::e
    SUITE("Suite of Case"):::d
    RUN("runner.py"):::c
    TASK("your task"):::u
    TRAJ("Trajectory"):::d
    EVAL("evaluators/"):::c
    SCORE("Score"):::d
    AGG("aggregate.py"):::c
    REP("Report"):::d
    OUT("console · runs/ · compare"):::o

    LIB --> RUN
    CLI --> RUN
    SUITE --> RUN
    RUN --> TASK --> TRAJ --> EVAL --> SCORE --> AGG --> REP --> OUT

    classDef e fill:#ede9fe,stroke:#8b5cf6,color:#3b0764,rx:8,ry:8
    classDef u fill:#fef3c7,stroke:#f59e0b,color:#451a03,rx:8,ry:8
    classDef c fill:#ccfbf1,stroke:#14b8a6,color:#042f2e,rx:8,ry:8
    classDef d fill:#e0f2fe,stroke:#38bdf8,color:#082f49,rx:8,ry:8
    classDef o fill:#ffe4e6,stroke:#fb7185,color:#4c0519,rx:8,ry:8

There are two ways in. From Python you call run, and on the command line you point Beetle at your own code with module:attr, e.g. --task mymod:agent. A path works too, so --task ./evals/agent.py:run is fine.

The task runs once per trial and every evaluator then judges that one recorded result, so adding an evaluator never means running the task again. Results come back in suite order no matter how they were scheduled, which is what makes two runs comparable line by line.

A case with no explicit id derives one from a hash of its input, so the same case keeps the same id across runs and across machines. Random ids would make two runs of one suite incomparable. Duplicate ids inside a suite are rejected at construction rather than at the point where they would quietly corrupt a comparison.

The built-in evaluators

All of them live in beetle.evaluators and score in [0.0, 1.0] unless a rubric says otherwise.

They are deliberately unforgiving about their own configuration. An unset expectation raises rather than quietly passing, an empty needle is rejected rather than matching everything, and a non-string output fails with its type named rather than being coerced through str(), where "42" would happily be found inside the integer 4200.

[!TIP] Every evaluator in a run needs a distinct name, because the name is the key its summary is reported under. Two bare Contains(...) instances both report as Contains and are rejected before the run starts. Pass name= to tell them apart: Contains("ok", name="has_ok").

Scoring how the system worked

Return a Trajectory instead of a bare value when the answer alone is not the thing you care about.

from beetle import Case, Step, ToolCall, Trajectory, run
from beetle.evaluators import CommandSucceeds, ToolsCalled


def agent(case):
    calls = (ToolCall("search", {"query": case.input}),)
    return Trajectory(
        final_output="done", steps=(Step("assistant", "", calls),), workspace="/tmp/run-42"
    )


run(
    [Case("fix the failing test")],
    agent,
    [ToolsCalled(("search",)), CommandSucceeds(["pytest", "-q"], timeout=60)],
)

That last pair is how you score a coding agent. Beetle never creates or cleans up the workspace, because your task owns that and simply reports where it worked. Workspace evaluators only read inside the directory they were given: an absolute path, or one climbing out through .., is a configuration error rather than a quietly satisfied assertion about a file the agent never touched.

[!WARNING] CommandSucceeds is a scoring primitive and not a sandbox. It runs exactly what you configured with the privileges of the process running Beetle, so keep untrusted input away from it. It takes an argument list rather than a shell string, so nothing in a case can be read as shell syntax, and on timeout it kills the whole process group. See SECURITY.md.

LLM judges

Beetle ships no provider client and makes no network calls of its own. A judge accepts any object with complete(prompt, system=...), so you write a short adapter for whatever you already use (see examples/model_adapter.py) and keep control of credentials, retries and model choice.

Judges answer with a named rubric level rather than a free-form number, since asking a model for a calibrated decimal invites it to drift onto whatever scale it feels like. Case data is fenced inside markers derived from its own content, so an output containing instructions cannot break out into the instruction channel. A refusal, an unparseable reply, or a model that raises an exception all produce an errored score, which is counted separately and left out of every mean.

from beetle.evaluators import Level, LLMJudge, Rubric

concise = Rubric(
    question="Is the answer as short as it can be while still answering the question?",
    levels=(
        Level("padded", 0.0, "Restates the question, hedges, or adds filler."),
        Level("acceptable", 0.5, "Mostly direct, with some unnecessary scaffolding."),
        Level("tight", 1.0, "Answers and stops."),
    ),
)

judge = LLMJudge(model=my_model, rubric=concise, pass_at=1.0, name="concise")

A rubric needs at least two levels, unique labels, and level names a verdict line can actually say — all checked when you build it, not when the bill arrives. abstain is reserved, so a judge that cannot grade the material says so instead of guessing.

[!TIP] FakeModel returns scripted replies, so you can exercise judge wiring without a provider. Beetle's own suite uses it, which is why the tests need no network and no keys.

Repeats, flake and intervals

Agents are not deterministic, and one trial per case cannot tell a fixed regression apart from a coin flip. repeat runs every case several times and reports how often the trials disagreed with each other.

report = run(suite, agent, [Contains("the answer is")], repeat=5)

run 20260818T120705Z-39efad03  (2 case(s) x 5 trial(s))
beetle 0.1.0 | python 3.12.12 | seed 0 | concurrency 1

cases: 8 passed, 2 failed, 0 errored/timed out  (pass rate  80.0% of judged)

evaluator                      pass     n  err  mean
Contains                      80.0%    10    0   0.800 [0.49, 0.94]  flake 20%

2 case(s) did not pass:
  FAIL     wobbly trial 2
           Contains = 0.00  'the answer is' not present in output
  FAIL     wobbly trial 4
           Contains = 0.00  'the answer is' not present in output

The bracket after the mean is a 95% interval: a Wilson score interval for binary outcomes, which behaves sensibly at small n and at rates of 0 or 1, and a seeded percentile bootstrap for continuous scores, so the same run always reports the same interval. flake is the mean per-case disagreement across trials, and a trial that errored before an evaluator could run still counts as a trial — ignoring it would report a case that fails half the time as perfectly stable.

This is also what keeps compare honest. A drop is only called significant when the two confidence intervals do not overlap, because reporting 0.90 → 0.85 on twelve cases as a regression trains people to ignore the tool.

Command line

beetle run cases.jsonl --task mymod:agent --evaluators mymod:evaluators \
    --repeat 5 --concurrency 8 --out runs/today --fail-under 0.9

beetle compare runs/yesterday runs/today
beetle report runs/today

--task and --evaluators take module:attr, resolved the way an import statement would. --evaluators accepts a single evaluator, a list of them, or a zero-argument factory returning either, which is how you build a judge that needs a configured client.

A .jsonl suite holds one case per line, and .json works too, either as a list or as an object with a cases key. Unknown fields are rejected rather than dropped, so a typo in expected is caught instead of silently unsetting the expectation.

{"id": "en", "input": "en", "expected": "hello"}
{"id": "fr", "input": "fr", "expected": "bonjour"}

Exit codes are a contract, because CI needs to tell a regression apart from a broken command. 0 means every gate was met, 1 means the run finished but a gate failed, 2 means bad usage or unreadable input, and 3 means an unexpected internal error. compare names the cases that newly fail and only calls a change significant when the two confidence intervals do not overlap.

evaluator                     baseline  candidate    delta
Contains                        100.0%      75.0%   -25.0%

* = intervals do not overlap; other movement is within noise

newly failing (1): gamma

REGRESSION

Runs on disk

--out runs/today, or write_run(report, "runs/today") from Python, leaves three files:

report.json is a public contract. Fields are added, never repurposed, schema_version is bumped only on a breaking change, and a version Beetle does not recognise is refused rather than guessed at. Reading a run never unpickles and never evaluates anything; it is JSON in, dataclasses out.

Re-running only what did not pass is a suite operation, not a flag:

first = run(suite, agent, evaluators)
retry = run(suite.select(first.unresolved_case_ids()), agent, evaluators)

unresolved_case_ids() is everything that did not pass — failures, errors and timeouts alike, since all three are worth another look and none of them is a result you would ship.

Execution model

The public API is synchronous. run awaits a coroutine task internally, so there is no second async surface to learn. Concurrency is a single parameter, max_concurrency, which defaults to 1. timeout and retries are arguments to the run, applied to every case in it.

[!IMPORTANT] Neither is guessed for you, and with no timeout a case runs until it returns, so set one whenever a task can hang.

A case that overruns its timeout is recorded as TIMEOUT and the run carries on. Python cannot kill a thread, so the case is abandoned rather than stopped. A replacement worker starts at once so the abandoned thread cannot hold a concurrency slot, and workers are daemon threads so it cannot keep the interpreter alive either. It does keep running until it returns on its own, which is why genuinely runaway work should be bounded inside the task, the way CommandSucceeds bounds it by killing its subprocess group.

Nothing in a task or an evaluator can end a run. Both are called inside a boundary that turns an exception into a recorded outcome against the case it happened in — KeyboardInterrupt and SystemExit excepted, since those are the operator asking to stop.

Examples

Every file under examples/ runs offline with no keys, and CI executes all of them on every push.

python examples/quickstart.py

beetle run examples/cases.jsonl \
    --task examples/suite.py:task \
    --evaluators examples/suite.py:evaluators

What Beetle leaves out

Chaos and fault injection, red teaming, simulated users, trace ingestion from observability backends, dataset generation and any hosted dashboard. Each of those is a separate product that happens to sit near evaluation, and folding them in would cost the small surface that makes this one worth reading end to end.

The way to add one is a package that imports Beetle, which needs nothing from us: a task is any callable taking a Case, and an evaluator is any object with evaluate(case, result).

Contributing

Issues and pull requests are welcome. CONTRIBUTING.md covers the setup, the four gates CI enforces — zero runtime dependencies, no .py file over 200 lines, mypy --strict, and a test suite that fails if any test opens a socket — and the checklist a new evaluator has to pass. Participation is governed by the Code of Conduct.

Security

Please do not report vulnerabilities in the issue tracker. SECURITY.md has the reporting process and the threat model, including what is deliberately out of scope: CommandSucceeds is not a sandbox, --task imports what you point it at, and a judge verdict is advisory rather than an authorisation decision.

Citation

@software{beetle,
  title    = {Beetle: TinyEvals for your AI agents and LLM/SLM apps},
  author   = {Banerjee, Pratyay},
  year     = {2026},
  url      = {https://github.com/Neilblaze/beetle},
  license  = {Apache-2.0},
  version  = {0.1.0}
}

GitHub's "Cite this repository" button reads CITATION.cff, which is the same metadata in machine-readable form.

License

Apache 2.0. See LICENSE.