Evalkeep

Stop fixing the same agent bug twice. Evalkeep turns production failures into a small, reviewed regression suite, and tells you whether a fix held — or that the evidence is too thin to say.

Existing eval tools execute tests. The hard part is deciding which of thousands of production traces deserve permanent coverage. Evalkeep owns that decision:

trace → failure evidence → failure family → representative case
      → reviewed regression test → runner execution → trustworthy comparison

Evalkeep does not run your agent and is not an eval framework. It generates tests, delegates execution to Promptfoo, and compares baseline against candidate. It sits upstream of your eval runner, not next to it.

Install

uv tool install evalkeep     # or: pipx install evalkeep, pip install evalkeep

Python 3.11+. Node.js is needed only for evalkeep run, which shells out to Promptfoo. The examples ship inside the package, so everything below works from a fresh install with no clone.

Quick start

One command takes a trace file to a review queue. Offline, no API key:

evalkeep demo .
evalkeep init
evalkeep from-traces refund-agent/traces.jsonl

5 trace(s) ingested
3 failure(s) found  explicit_status x3, failed_evaluator x1, negative_feedback x2
2 failure famil(ies)
3 with enough evidence for a regression test
3 of them only forbid the mistake that was observed; say what should have happened at review.

Review them: evalkeep review  (3 pending)

That second qualifier is the honest state of a first run: nothing in a trace says what the agent should have done. Describing the failures closes it, by hand at review or with an analyzer configured. It stops at review on purpose — approving a test is a judgement.

On real agent data

The example above is five traces. Here is the same pipeline on tau-bench trajectories: 165 retail and airline customer-service tasks per model, each scored by comparing the final database state against the expected one. Real tool calls, and an independent verdict — the two halves a regression suite needs.

python tau-bench/prepare.py                      # ~8 MB, two models
evalkeep from-traces tau-bench/Qwen3-235B-A22B-FP8.traces.jsonl

165 trace(s) ingested
90 failure(s) found  explicit_status x90, failed_evaluator x90
4 failure famil(ies)

Four families, from 90 failures: retail exchange and refund flows, airline reservation changes, and two shapes of giving up and escalating to a human. Build a test per failure, approve them, and run two recorded models against the suite one of them produced:

evalkeep dataset build --all
evalkeep review
evalkeep targets add baseline  --type python --function call_api \
  --path tau-bench/replay_Qwen3_235B_A22B_FP8.py
evalkeep targets add candidate --type python --function call_api \
  --path tau-bench/replay_claude_4_5_sonnet_thinking_off.py
evalkeep run --target baseline && evalkeep run --target candidate
evalkeep compare

compared                           89
baseline pass rate               4.5%
candidate pass rate             42.7%
difference                     +38.2%
p-value                        0.0000
95% interval         +27.2% to +49.2%
McNemar's exact test: the change is unlikely to be chance.

1 test(s) excluded and not counted in any rate above.

Baseline scoring 4.5% is the control: the tests came from its own failures, so it should fail nearly all of them. The four it passes are the documented weakness of deriving a test with nothing describing the failure — assertions target the last tool call, which is sometimes a harmless lookup. The excluded test is one whose target raised rather than answered, and it is kept out of every rate rather than counted as a failure.

Doing it stage by stage

from-traces runs five commands in order. Each is a real decision with its own evidence, and you will want them separately once you are tuning a suite:

evalkeep ingest traces.jsonl   # validate, redact, store
evalkeep detect                # evidence-backed failures
evalkeep analyze               # describe them (or: failures label)
evalkeep discover              # embed, cluster, pick representatives
evalkeep dataset build         # draft a test per representative

Re-running from-traces is safe: it skips traces it already has and rebuilds drafts, but never touches a test you have reviewed.

Bring your own traces

evalkeep ingest opentelemetry/spans.json --format otlp   # OpenTelemetry / OpenInference
evalkeep ingest langsmith/runs.jsonl --format langsmith  # LangSmith

Those paths are what evalkeep demo writes — the same five interactions exported from each tool, so you can see what an adapter does before pointing one at your own data. Adapters read files, never APIs: no credentials, any vendor tier. OpenTelemetry covers the most ground, since Langfuse, Braintrust and Phoenix all ingest OTLP.

Commands

What it guarantees

• Values are redacted before storage — in memory, with no path around it. Identifiers can be pseudonymized too.
• Automation never overwrites human judgement. Re-running any stage refreshes derived data and leaves your reviews, labels and edits alone.
• Nothing is exported without approval. Generated tests are drafts.
• A test that never ran is not a test that failed. Timeouts and crashed providers are excluded from every rate, so an outage cannot read as a regression.
• One lucky pass is not a fix. run --repetitions N reports a per-case verdict; a case that only sometimes passes is flaky, never passing.
• Score changes are not overclaimed. McNemar's exact test, and no confidence interval when the sample cannot support one.

More

How it works · Privacy and security · Roadmap · Contributing · Changelog

git clone https://github.com/rakshita-devurkar/evalkeep && cd evalkeep
uv sync && uv run pytest              # 985 tests
uv run ruff check . && uv run mypy    # lint and strict types

EVALKEEP_E2E=1 uv run pytest also runs the suite against real Promptfoo. Exit codes: 0 success, 1 ran but some records were rejected, 2 could not run.

0.1 is feature-complete. Multi-turn replay, longitudinal failure history, and clustering that does not hold the whole distance matrix in memory are still open; see the roadmap. Pre-1.0, minor versions may break the CLI and the on-disk layout — see the changelog.

License

Apache-2.0. See LICENSE.