EvalShift

Run your prompts on two LLMs and find out, with statistical confidence, what regressed.

EvalShift is a local-first CLI that helps engineering teams migrate safely between LLM versions (for example gemini-2.5-flash → gemini-3.1-flash-lite-preview). Point it at your prompts and a golden suite of inputs; it runs both models, scores the outputs with structural / semantic / LLM-as-judge / tool-call evaluators, and produces a single-file HTML report with defensible statistics: paired tests, Cohen's d, 95% CIs, and Benjamini-Hochberg correction across every (prompt x evaluator x slice) comparison.

Local runs stay on your machine by default. Hosted private-alpha commands are available when you explicitly log in and push a run.

Status

Alpha. Every command in the pipeline is shipped and the test suite is at 95%+ coverage. APIs may still change as feedback comes in.

Install

Requires Python 3.14+.

# Recommended
uv pip install evalshift     # or: pip install evalshift

From source (for contributors):

git clone https://github.com/babaliauskas/evalshift-cli.git
cd evalshift-cli
uv venv --python 3.14
source .venv/bin/activate
uv pip install -e ".[dev]"

Quick start

The fastest first run costs nothing and needs no API keys:

# 1. Scaffold the runnable demo project. Writes evalshift.yaml + prompts.py +
#    tools.yaml + golden.jsonl + fixtures.jsonl for a support agent.
mkdir evalshift-demo && cd evalshift-demo
evalshift demo

# 2. Run the whole pipeline deterministically, without API keys.
evalshift all --offline --yes --open

# 3. For a live run, set whichever provider keys you'll use.
export GEMINI_API_KEY=<gemini-api-key>   # or ANTHROPIC_API_KEY / OPENAI_API_KEY

# 4. Run the live pipeline in one command (doctor → run → evaluate
#    → analyze → report). Pass --open to launch the report.
evalshift all --yes --open

For a real project, start from evalshift init instead — it writes a single minimal, capture-first evalshift.yaml. Instrument your agent with the companion evalshift-sdk so it records captures, then promote them into golden suites:

evalshift init                    # minimal capture-first config
# ... run your instrumented agent with EVALSHIFT_CAPTURE=1 ...
evalshift capture sync            # captures → golden suites + wired config
evalshift all --suite-name <suite> --to <candidate-model>

evalshift all drives the full five-stage pipeline under a single Rich Live region — stacked status rows, an inline progress bar for the run stage, and a final verdict block that tells you whether the candidate is significantly better, regressed, or showed no significant change.

If you want to drive each stage by hand (useful in CI, or when re-running just one stage after fixing config):

evalshift doctor
evalshift run --yes
evalshift evaluate <run-id>
evalshift analyze <run-id>
evalshift report <run-id> --open

Every artefact lives under .evalshift/runs/<run-id>/ — state.json, raw.jsonl, scores.jsonl, analysis.json, report.json, report.html. None of it leaves your machine unless you opt in to hosted upload commands.

Hosted private alpha

Hosted EvalShift adds shared run history, web viewing, diffs, and GitHub PR comments. It is optional: local CLI usage does not require an account.

# Sign in through the hosted web app, then approve CLI login in the browser.
# Defaults to https://api.evalshift.dev; pass --host to target another server.
evalshift login
evalshift whoami

# Add a hosted project to evalshift.yaml:
# project: acme/model-migration
# thresholds:
#   pass_rate_min: 0.95

# Run locally, then package and push the result.
evalshift all --yes --push

You can also drive the hosted steps manually:

evalshift bundle <run-id>
evalshift push <run-id>
evalshift push --bundle .evalshift/runs/<run-id>/run_bundle.json.gz

Credential precedence is explicit CLI flags, then EVALSHIFT_HOST / EVALSHIFT_TOKEN, then ~/.evalshift/credentials.

GitHub Action

evalshift init --ci scaffolds a workflow that runs EvalShift on pull requests, pushes the run to hosted EvalShift, compares against the latest compatible base-branch run, posts or updates one PR comment, and sets the evalshift/regression commit status.

Required setup:

evalshift init --ci

Then add repository secrets for EVALSHIFT_TOKEN and the provider keys your models use. The generated workflow uses:

uses: babaliauskas/evalshift-action@v0

See docs/github-action.md for workflow permissions, fail-on modes, and baseline behavior.

Agent migrations

Migrating an agent (a prompt that uses tools)? EvalShift detects regressions in which tools the new model calls, what arguments it passes, and how it sequences them. The killer scenario: a routing agent that silently stops calling notify_security_team after the migration — text-only eval reports green, EvalShift marks it CRITICAL.

The evalshift demo scaffold is an agent project — six tools, a 40-row golden suite, and deterministic replay fixtures. Just run the offline quick-start above and the tool-call evaluators plus migration policy verdict kick in automatically.

If your tool definitions already live in Python, evalshift tools sync <definitions.py> extracts them (AST-parsed, never imported) into .evalshift/tools.json and wires tools_path: onto your prompts.

See docs/agents.md for the full walkthrough and the examples/agent/ directory for a runnable customer-support example.

What the report looks like

Generate a deterministic example locally — no API keys required:

scripts/run_showcase.sh --offline --only pass-clean --open

That runs the examples/showcase/pass-clean/ scenario with the bundled fixtures.jsonl, writes a single-file HTML report under .evalshift/runs/<run-id>/report.html, and opens it in your browser.

The HTML report (single file, no external assets, works offline) has:

• Executive summary — one row per prompt with a severity badge.
• Per-prompt deep dive — aggregate stats, per-slice breakdown, top-5 worst regressions side-by-side.
• Methodology appendix — every test, p-value, effect size, and CI is documented.

Why local-first?

Your prompts and suite stay local for doctor, run, evaluate, analyze, and report. The only outbound calls in local mode are to the LLM providers you configure (Anthropic, OpenAI, Google) using your own API keys.

bundle packages completed local artifacts into run_bundle.json.gz without uploading them. push and all --push upload that bundle to the hosted backend associated with your token.

Documentation

• DOCS.md — consolidated single-file reference for everything below
• Getting started — install + first run walkthrough
• Configuration reference — every evalshift.yaml field
• Evaluators — when to use which family
• Agent migrations — tool-call evaluation, tools sync
• Multi-turn conversations — teacher-forced replay
• Agent traces — bring-your-own agent timelines
• Methodology — the statistical machinery
• Hosted alpha — login, bundle, push, thresholds, privacy
• GitHub Action — PR comments + hosted regression gate
• FAQ — common questions
• llms-full.txt — dense single-file reference for AI coding tools, hosted at https://evalshift.dev/cli/llms-full.txt

Non-goals

• General-availability hosted service or billing
• Hosted provider-key storage
• Multi-criterion judge in a single call
• Custom evaluator plugin system
• Comparing more than 2 models in one run
• Auto-detection of LangChain / LlamaIndex prompt patterns

License

AGPL-3.0-or-later. Free for any use, including commercial, provided that derivative works — including network-hosted services — are released under the same license.

Versions 0.3.0 and earlier (published on PyPI before this change) remain available under the MIT License terms they were released with.

Commercial licenses without the AGPL share-back requirement are available; contact l.babaliauskas@gmail.com.