jev-harness

Test your AI coding-agent setup the way you test your code.

https://github.com/user-attachments/assets/58b43a69-6fed-4d2a-a5a4-372ab175dcdb

The 3-minute launch video (sound on): the problem, the architecture, and the four tools on real runs. Every number in it comes from those runs; the compaction part is a simulated run and is labelled so. How it was made: demo/launch-video.

You configure coding agents (Claude Code, Codex, Cursor) with instruction files, hooks, permission rules, AI code reviewers, and cheap model "gates". Almost nobody tests that configuration. jev-harness answers four questions about it:

Install

The package is jev-harness-sdk on PyPI; the command it installs is jev-harness. Python 3.11 or newer.

uv tool install 'jev-harness-sdk[stats]'      # recommended: an isolated tool install
pipx install 'jev-harness-sdk[stats]'         # same thing with pipx
pip install 'jev-harness-sdk[stats]'          # into the current environment

[stats] adds scipy for exact Clopper-Pearson bounds in calibrate; leave it out for a smaller install. An unrelated PyPI package called jev-harness installs a command and a Python package with the same names as this one, so don't install both into one environment (uv tool and pipx keep tools apart).

Quickstart

# 1. Set your Jev key (optional: every command works without a judge).
export JEV_API_KEY=...                       # or put JEV_API_KEY=... in a .env file in your project
jev-harness judge check                      # one real call: is the key and endpoint OK?

# 2. Run a check on your own agent setup, from your project folder.
jev-harness guard run --project . --format summary
jev-harness guard run --project . --judge jev --format summary     # with Jev's second opinion

Never commit your key: keep .env in .gitignore. Before testing against real agents, GitHub, or reviewer apps, read docs/REAL_TESTING.md. For development on this repository: uv sync --all-extras, then prefix commands with uv run.

Every command takes --format md|json|github|summary (summary is a short terminal readout), --output PATH, --fail-on flag|escalate, and --dry-run. Exit code 0 is clean, 1 means a finding reached --fail-on, and 2 means the harness could not run (bad input, missing credential, a budget was hit). Save a result with --format json and render it later with jev-harness report result.json --format md.

Five minutes per module

guard: do my guards stop disguised commands?

Your guards already exist: permissions and PreToolUse hooks in .claude/settings.json. This sends them dangerous commands in 11 disguises (r''m, X=rm; $X, base64, python -c, a file tool instead of a shell, ...) plus harmless controls, and lists every exact variant that got through. Nothing is executed.

jev-harness guard run --dry-run          # what would be tested
jev-harness guard run                    # the report; exit 1 if anything got through
jev-harness guard run --format json -o before.json
jev-harness guard run --compare before.json    # after changing a hook or upgrading the agent

Why it exists: permission wildcards and hand-written hooks are easy to get subtly wrong, and each hook tool tests only its own rules.

calibrate: can I trust my gate's confidence?

Give it a log of gate decisions (id, decision, confidence) and a person's labels.

jev-harness calibrate sheet gate.jsonl --out sheet.csv --n 100          # label this sample
jev-harness calibrate report gate.jsonl --labels sheet.csv --target 0.01
jev-harness calibrate drift gate.jsonl --labels sheet.csv --split 2026-09-01T00:00:00Z

You get a reliability table, the calibration error, the miss rate at each threshold with an upper bound, the cheapest threshold that stays under your target, and how many more labels you would need if the data cannot yet say.

Why it exists: a confidence threshold is usually picked by feel, and no one measures how often the "safe to skip" calls are wrong.

compaction: what does my agent forget?

jev-harness compaction run --dry-run     # the instruction sources found and how each one loads
jev-harness compaction run               # simulated, free: compares summarizers on your files
jev-harness compaction run --mode real --max-runs 80 --max-cost 5 --rounds 2 --trials 3

It plants a unique codeword in a copy of each instruction source, compacts, and reports which sources the agent still recalls, per round, with intervals. Simulated mode (the default) measures a summarizer, is labelled as such, and is not evidence about any agent. Real mode needs a spend cap.

Why it exists: instructions that vanish after compaction fail silently, and which ones vanish depends on where you put them.

reviewers: which AI reviewer is worth paying for?

jev-harness reviewers mine --repo .                         # candidate fixes from your history
jev-harness reviewers build --repo . --target-repo you/sandbox --candidates candidates.json --dry-run
jev-harness reviewers build --repo . --target-repo you/sandbox --candidates candidates.json
jev-harness reviewers collect --reviewer 'coderabbitai[bot]' --wait 900
jev-harness reviewers report
jev-harness reviewers cleanup

It re-introduces bugs your team already fixed as pull requests on a sandbox repository you name, plus bug-free control PRs, and scores each reviewer's recall, precision, noise, and latency by where its comments land. This pushes code to GitHub and every reviewer app on the sandbox will read it; read REVIEWERS first.

Why it exists: reviewer benchmarks are run by vendors on open-source code, not on yours.

In CI

action/action.yml is a composite GitHub Action for guard and calibrate:

- uses: Adityakk9031/jev-harness/action@v1.2.0
  with:
    command: guard
    args: run --project .
    judge: jev                                  # optional
    jev-api-key: ${{ secrets.JEV_API_KEY }}     # required when judge is jev

Using Jev as the judge

TypeSafe Jev is the judge this tool is built around. Four commands can use it, and every one works without it.

export JEV_API_KEY=...                      # or a .env in this folder, or --env-file ~/.jev.env
jev-harness judge check                     # one real call: does the key and endpoint work?

To use Jev wherever a command supports a judge, copy jev-harness.example.yml to jev-harness.yml (it sets judge: jev), or export JEV_JUDGE=jev. Otherwise pass --judge jev per command.

• A judge never overrides a rule. It adds an opinion next to a result; the deterministic result stands.
• Jev's confidence is not a probability of being right (it is how far the top probability sits above an even split). calibrate gate therefore logs the chosen option's probability, and docs/LIVE_RESULTS.md has what a several-thousand-call run on synthetic data showed.
• A failed or unsure judge is "no opinion" (below min_judge_confidence, default 0.7), never a confirmation or a pass.
• Every judge command has a call cap (--max-runs, default 200) checked before the first call, and --dry-run prints the number of calls. Judging sends payloads, diffs, and comments to the judge's service.
• Anthropic and OpenAI judges exist behind the same flag (--judge anthropic|openai, with the matching extra and key) but are unit-tested only. Jev is the one run live.

Design promises

1. It never touches your real setup: everything runs on a disposable copy.
2. It never executes a dangerous payload: payloads reach a guard as JSON on stdin, never a shell.
3. It never pushes to a repository you did not name and confirm.
4. Deterministic rules have the final say; a model can add findings and opinions but never override an escalate, and a judge that fails or is unsure escalates instead of passing.
5. A rate measured from repeated trials ships with its sample size and a confidence interval, and a small sample is reported as provisional rather than as a result.
6. No number in these docs or in a report comes from anywhere but a run of this tool on your data.

Honest limitations (1.1.0)

• Several paths have not been run live. No claude or codex CLI, GitHub sandbox repository, or real reviewer app was available while building this release. Agent behaviour is covered with scripted agents, GitHub with a mock plus a local bare repository, and the Action's script with a local uvx run. The first live run of compaction --mode real and of reviewers build is the first test of those paths; please report what you see.
• guard approximates permission rules. Hooks are executed for real; permissions are matched by an emulation (which understands quoting and top-level operators but not $(), bash -c, or interpreters) and the report says so. Confirm any permission bypass in the real agent.
• compaction real mode relies on an unverified trigger prompt (/compact by default), confirmed per measurement point by a PreCompact hook. If it cannot confirm a compaction it stops with exit 2 and reports no rate. Real mode supports Claude Code only; Codex sources work in simulated mode.
• reviewers scores by comment location, which shows a reviewer looked at the bug, not that it explained it. Each reviewer sees each bug once, so one run is one sample.
• Judge opinions are advisory. They are only as good as the model, they were checked live on a handful of items (not benchmarked), and a judge call sends your payloads, diffs, and comments to the judge's service.
• Skills and subagents are not probed by compaction; user-level instruction files are out of scope.
• A clean report means the payloads, labels, or bugs you gave it passed, not that your setup is safe.

Development

See CLAUDE.md for the commands, docs/PLAN.md for the plan, decisions, and as-built notes, docs/AGENT_CLIS.md for the agent flags this relies on and whether each has been observed, and docs/ROADMAP.md for what was deferred.

uv sync --all-extras
uv run ruff check . && uv run ruff format --check . && uv run mypy && uv run pytest

MIT licensed.