EvalShift

Open-source LLM migration and regression testing for AI agents.

The SDK captures what your agent really did. The CLI replays it against a candidate model and tells you what broke. The hosted app keeps the history.

evalshift-sdk    captures real agent behavior in production
      ↓
evalshift CLI    replays it against a candidate model — scores, stats, report
      ↓
hosted (opt-in)  run history, diffs, PR gates

Migrating between LLM versions (say gemini-2.5-flash → gemini-3.1-flash-lite-preview) means guessing which behaviors changed. EvalShift removes the guess. It runs both models over the same golden suite, scores the outputs with structural / semantic / LLM-as-judge / tool-call evaluators, and produces a single-file HTML report with defensible statistics: paired tests, Cohen's d, 95% CIs, and Benjamini-Hochberg correction across every (prompt x evaluator x slice) comparison.

An eval is only worth the examples in it. That is why the capture SDK is part of the product rather than an add-on: it records real production runs — model calls, tool calls, final outputs — to disk, and evalshift capture sync promotes them into golden suites. Hand-written suites are fully supported too, but captured traffic is the recommended starting point.

Local runs stay on your machine by default. Hosted private-alpha commands are available when you explicitly log in and push a run.

How EvalShift fits together

Four pieces, released and documented independently:

The SDK and the CLI never call each other — the interface is files under .evalshift/captures/, so either works without the other. Because both use the top-level import name evalshift, install them in separate virtual environments: the SDK in your agent's, the CLI wherever you run evaluations.

For AI coding agents

Point your coding agent at the dense, single-file reference for the piece it is working on:

• EvalShift CLI: https://www.evalshift.dev/cli-llms-full.txt (source of truth: llms-full.txt in this repo)
• EvalShift SDK: https://www.evalshift.dev/sdk-llms-full.txt
• EvalShift GitHub Action (CI): https://www.evalshift.dev/ci-llms-full.txt

Status

Alpha. Every command in the pipeline is shipped and the test suite covers 92% of the source. APIs may still change as feedback comes in.

Install

Requires Python 3.11+.

# Recommended
uv pip install evalshift     # or: pip install evalshift

And, in your agent's virtualenv — a separate one, see above — the capture SDK that feeds the CLI its suites:

uv pip install evalshift-sdk     # or: pip install evalshift-sdk

From source (for contributors):

git clone https://github.com/babaliauskas/evalshift-cli.git
cd evalshift-cli
uv venv --python 3.11
source .venv/bin/activate
uv pip install -e ".[dev]"

Quick start

Use it on your agent

This is the workflow: record what your agent really does, then hold a candidate model to it.

evalshift init                    # minimal capture-first evalshift.yaml

Instrument the agent with evalshift-sdk — installed in the agent's own virtualenv, stdlib-only, Python 3.10+:

from evalshift import capture


@capture.tool(name="issue_refund")
def issue_refund(order_id: str) -> dict: ...


@capture.agent(suite="support_agent", redact=True)
def handle(message: str) -> str: ...

redact= is required on every entry point that opens a capture session (SDK 0.3.0+) — @capture.agent, capture.agent_session, capture.agent_session_async, EvalShiftCallbackHandler: True masks emails, API keys and bearer tokens before anything reaches disk, False records verbatim, or pass your own (value) -> value callable. @capture.tool takes no redact of its own — tool spans are masked by the redactor of the agent session they run inside.

Nothing is recorded unless EVALSHIFT_CAPTURE=1 is set, so the decorators are safe to leave in production permanently:

EVALSHIFT_CAPTURE=1 python your_agent.py   # writes .evalshift/captures/
evalshift capture sync                     # captures → golden suites + wired config
evalshift all --suite-name support_agent --to <candidate-model>

See docs/sdk.md for the full capture contract. Can't instrument the agent? A hand-written golden.jsonl works just as well — see Getting started.

Driving the pipeline

evalshift all drives the full five-stage pipeline under a single Rich Live region — stacked status rows, an inline progress bar for the run stage, and a final verdict block that tells you whether the candidate is significantly better, regressed, or showed no significant change.

If you want to drive each stage by hand (useful in CI, or when re-running just one stage after fixing config):

evalshift doctor
evalshift run --yes
evalshift evaluate <run-id>
evalshift analyze <run-id>
evalshift report <run-id> --open

Every artefact lives under .evalshift/runs/<run-id>/ — state.json, raw.jsonl, scores.jsonl, analysis.json, migration_decision.json (when the config sets a migration_policy), report.json, report.html, and insights.json (when insights ran). None of it leaves your machine unless you opt in to hosted upload commands.

Hosted private alpha

Hosted EvalShift adds shared run history, web viewing, diffs, and GitHub PR comments. It is optional: local CLI usage does not require an account.

# Sign in through the hosted web app, then approve CLI login in the browser.
# Defaults to https://api.evalshift.dev; pass --host to target another server.
evalshift login
evalshift whoami

# Add a hosted project to evalshift.yaml:
# project: acme/model-migration
# thresholds:
#   pass_rate_min: 0.95

# Run locally, then package and push the result.
evalshift all --yes --push

You can also drive the hosted steps manually:

evalshift bundle <run-id>
evalshift push <run-id>
evalshift push --bundle .evalshift/runs/<run-id>/run_bundle.json.gz

Credential precedence is explicit CLI flags, then EVALSHIFT_HOST / EVALSHIFT_TOKEN, then ~/.evalshift/credentials.

What gets uploaded

Nothing, until you run push (or all --push) — and the CLI itself has no telemetry, analytics, or crash reporting. A push uploads one file, run_bundle.json.gz, whose full field-by-field contract is documented in docs/hosted.md — Privacy model. The short version:

• Uploads: the run manifest (model ids, suite name, git SHA/branch/PR number, content hashes, CLI version); per-example rows — the example's template inputs and expected output verbatim, both models' full outputs, tool-call traces (names and arguments), scores, cost and latency; aggregate statistics, the analysis, the migration decision, economics, and the machine-written insights narrative.
• Never uploads: provider API keys, prompt bodies and system prompts, suite conversation histories, tool definitions/schemas, raw.jsonl, the response cache, captures, and report.html. Prompt and dataset content is replaced by SHA-256 hashes so diffs still align across runs.
• Can still be sensitive: inputs, expected outputs, model outputs, and traces carry whatever content your suite or your models put in them. Redact at capture time (see the SDK's redaction boundary) and inspect before pushing: evalshift bundle <run-id> writes the exact bytes a push would upload — gunzip -c .evalshift/runs/<run-id>/run_bundle.json.gz | jq ..

GitHub Action

evalshift init --ci scaffolds a production-shaped workflow: it discovers every committed suite under .evalshift/suites/, evaluates each on every pull request via babaliauskas/evalshift-action@v0 (one matrix job per suite), pushes the runs to hosted EvalShift, compares against the latest compatible base-branch run, posts one PR comment, and gates merges on your migration_policy through a single required evalshift gate check. The full setup checklist — secrets, committing suites, branch protection — is documented in the generated file itself.

evalshift init --ci

Then add repository secrets for EVALSHIFT_TOKEN and the provider keys your models use; until they exist the workflow no-ops green with a notice.

See docs/github-action.md for the workflow's shape, fail-on modes, and baseline behavior.

Agent migrations

Migrating an agent (a prompt that uses tools)? EvalShift detects regressions in which tools the new model calls, what arguments it passes, and how it sequences them. The killer scenario: a routing agent that silently stops calling notify_security_team after the migration — text-only eval reports green, EvalShift marks it CRITICAL.

Each golden-suite example carries its own toolset — recorded automatically by capture promote / capture sync from your production captures, or inlined by hand for a hand-authored suite.

See docs/agents.md for the full walkthrough and the examples/agent/ directory for a runnable customer-support example.

What the report looks like

Every run writes a single-file HTML report to .evalshift/runs/<run-id>/report.html — see Use it on your agent above, or the runnable walkthrough in examples/agent/. The report (single file, no external assets, works offline) has:

• Migration verdict — the policy decision up top: which budgets failed, the top regression causes, and the recommendation.
• Executive summary — one row per prompt with a severity badge.
• What changed, in plain language — the verdict, the economics and the behavioural drift explained by defaults.insights_model. Every figure in it is copied from the computed statistics, never generated. Needs a provider key; skip it with --no-insights.
• Per-prompt deep dive — aggregate stats, per-slice breakdown, top-5 worst regressions side-by-side.
• Methodology appendix — every test, p-value, effect size, and CI is documented.

Why local-first?

Your prompts and suite stay local for doctor, run, evaluate, analyze, and report. The only outbound calls in local mode are to the LLM providers you configure (Anthropic, OpenAI, Google) using your own API keys.

bundle packages completed local artifacts into run_bundle.json.gz without uploading them. push and all --push upload that bundle to the hosted backend associated with your token.

Wiring the agent references into your project

The three references are listed at the top of this README. evalshift init wires these links into your project automatically: it writes EVALSHIFT.md and points existing agent files (AGENTS.md, CLAUDE.md, GEMINI.md, .cursorrules, .github/copilot-instructions.md) at it, creating AGENTS.md if none of those files exist. Disable with --no-wire-agents. In this repo the same three links live in AGENTS.md.

Documentation

• DOCS.md — consolidated single-file reference for everything below
• Getting started — install + first run walkthrough
• Configuration reference — every evalshift.yaml field
• Evaluators — when to use which family
• Agent migrations — tool-call evaluation, per-example toolsets
• Multi-turn conversations — teacher-forced replay
• Agent traces — bring-your-own agent timelines
• Capture SDK — instrument your agent, promote captures to suites
• Methodology — the statistical machinery
• Hosted alpha — login, bundle, push, thresholds, and the privacy model: exactly what data uploads and what never leaves your machine
• GitHub Action — PR comments + hosted regression gate
• FAQ — common questions
• llms-full.txt — dense single-file reference for AI coding tools, hosted at https://www.evalshift.dev/cli-llms-full.txt

Non-goals

• General-availability hosted service or billing
• Hosted provider-key storage
• Multi-criterion judge in a single call
• Custom evaluator plugin system
• Comparing more than 2 models in one run
• Auto-detection of LangChain / LlamaIndex prompt patterns

License

AGPL-3.0-or-later. Free for any use, including commercial, provided that derivative works — including network-hosted services — are released under the same license.

Versions 0.3.0 and earlier (published on PyPI before this change) remain available under the MIT License terms they were released with.

Commercial licenses without the AGPL share-back requirement are available; contact l.babaliauskas@gmail.com.