orvix-provepr

Your coding agent says it's done. provepr makes it prove it.

Before an agent (Claude Code, Codex, Cursor, or anything that reads AGENTS.md) calls a change done or opens a PR, it writes down what the change claims and how to check it. provepr runs those checks in a real browser, against the app's HTTP API and as shell commands, and hands back facts: which step failed, HTTP 5xx responses, page and console errors, command output, a screenshot after every step, and a video. The agent fixes what failed and runs the checks again. By the time a human opens the PR, each claim is marked PASS, FAIL or UNVERIFIED.

provepr never calls an LLM. The agent's own model writes the plan and reads the evidence. provepr only runs the browser and reports what happened.

$ provepr run
# provepr run 20261018-042535-318161
1 PASS · 1 FAIL · 1 UNVERIFIED  (base_url http://127.0.0.1:8765)

## PASS `page-loads`: The profile page loads with a Name field and a Save button
- steps passed: 4/4

## FAIL `save-persists`: Clicking Save stores the name, and it is still there after a reload
- steps passed: 5/6
- FAILED step 6 `expect_value {"label": "Name", "value": "Ada Lovelace"}`: Locator expected to have Value 'Ada Lovelace' | Actual value:
- HTTP 500 POST http://127.0.0.1:8765/api/profile

## UNVERIFIED `name-length-limit`: The API rejects names longer than 100 characters
- no runnable steps (not checkable in a browser yet)

In that run the page showed "Saved" while the server returned a 500, so the bug would have shipped. With provepr set up, Claude Code found and fixed it by itself (see What it caught).

Quick start

uv tool install orvix-provepr      # or: pipx install orvix-provepr
cd your-app && provepr init

provepr init sets the repo up so every agent and CI use provepr the same way:

Running it again changes nothing. Existing files are kept, and only the marked AGENTS.md block is refreshed. After that, just ask your agent for changes as usual.

How it works

 Coding agent (Claude Code / Codex / anything that reads AGENTS.md)
   │ 0. AGENTS.md says: read `provepr docs` (the guide bundled with the installed version)
   │ 1. writes the change + adds claims to .provepr/plan.yaml (the steps that prove them)
   │ 2. runs `provepr check`, then `provepr run` (or the run_proof MCP tool)
   ▼
 provepr
   ├─ starts the app (optional `start:` command) and waits for it
   ├─ runs each claim in headless Chromium: video, a screenshot after every step,
   │  console errors, page errors, every HTTP 4xx/5xx and failed request,
   │  plus `http` steps that call the app's API directly, `mock` steps that fake
   │  failing or slow requests, and `cmd` steps that run shell commands
   ├─ marks each claim PASS / FAIL / UNVERIFIED using fixed rules, no AI
   └─ writes the evidence bundle to .provepr/runs/<run>/
   ▼
 Agent reads a short report (about 250 tokens) and one contact sheet image
   └─ get_frames(claim) for screenshots of the failing step → fixes the code → runs again

What counts as a failure. A claim FAILs if a step fails, the page throws an uncaught error, a request returns HTTP 5xx, or a request fails at the network level (requests a mock step faked on purpose are left out). Console errors, HTTP 4xx responses and visual changes are listed for the agent to judge but do not fail the claim. A claim with no steps is UNVERIFIED, never PASS.

Why the plan comes first. The agent decides what "working" looks like before it sees any evidence, and the guide forbids weakening the plan to make it pass. An agent that grades its own work this way is much harder to fool, including by its own optimism.

Why the plan accumulates. Each change adds claims and keeps the old ones, so plan.yaml grows into a regression suite as a side effect, and CI runs all of it. provepr check --base main enforces it without AI: a PR that removes a claim, or removes or changes one of its checks, fails unless the plan gives a reason under waivers:, which then shows up in the PR diff.

How agents learn the rules. The guide for coding agents ships inside the package (src/provepr/docs/guide.md), so it always matches the installed version. Agents reach it through provepr docs, the get_guide MCP tool, or the plugin's skill. All three point to the same file. provepr check enforces the parts a guide can't: unknown steps, unsafe ids, and claims with no checks are rejected before anything runs.

Install

You need uv. On first run provepr downloads Chromium for Playwright by itself.

Any agent (AGENTS.md)

uv tool install orvix-provepr
provepr init

This works with any agent that reads AGENTS.md and can run shell commands.

Claude Code

/plugin marketplace add orvixx-labs/orvix-provepr
/plugin install provepr@orvix

To load it for a single session from a local checkout: claude --plugin-dir /path/to/orvix-provepr.

Codex

codex plugin marketplace add orvixx-labs/orvix-provepr
codex plugin add provepr

Or add it as a plain MCP server:

codex mcp add provepr -- uvx --from orvix-provepr provepr mcp

Codex support has not been tested end to end yet. Claude Code has (see below). Please open an issue if it doesn't work for you.

Use it

Ask the agent to prove its change works before it opens a PR, for example:

Before you open the PR, prove the feature works with provepr and fix anything that's broken.

With provepr init done or the plugin installed, the agent learns the plan format and the loop from the bundled guide: claims first, then check, run, read, fix, and run again.

Writing a plan

# .provepr/plan.yaml  (commit this; ignore .provepr/runs/)
base_url: http://localhost:3000
start: npm run dev          # optional; run from the project root, stopped after the run

claims:
  - id: save-persists
    says: "Clicking Save stores the name, and it is still there after a reload"
    steps:
      - goto: /settings
      - fill: { label: Name, value: Ada }
      - click: { role: button, name: Save }
      - expect_text: Saved
      - reload: true
      - expect_value: { label: Name, value: Ada }

  - id: name-too-long-rejected
    says: "The API rejects names longer than 100 characters"
    steps:
      - http: { method: POST, path: /api/profile, json: { name: "..." }, expect_status: 400 }

  - id: save-failure-shown
    says: "When saving fails, the page says so instead of 'Saved'"
    steps:
      - goto: /settings
      - mock: { path: /api/profile, method: POST, status: 500 }
      - click: { role: button, name: Save }
      - expect_text: Could not save

  - id: cli-export
    says: "`mytool export` writes the profile as JSON"
    steps:
      - cmd: { run: mytool export --json, expect_output: '"name"' }

The full step and locator reference is in the guide: run provepr docs, or read src/provepr/docs/guide.md.

MCP tools

Command line

provepr init [dir] [--no-ci]                  # set up AGENTS.md, CLAUDE.md, plan, CI workflow, .gitignore
provepr docs                                  # print the guide for coding agents
provepr check [--plan PATH] [--base REF]      # validate the plan (and that nothing was removed since REF)
provepr run [--plan PATH] [--base-url URL] [--only ids] [--keep-evidence] [--save-baseline] [--compare]
                                              # exits 1 on any FAIL
provepr mcp                                   # serve the MCP tools over stdio

base_url resolves in this order: --base-url, then the PROVEPR_BASE_URL environment variable, then the plan.

The evidence bundle

.provepr/runs/20261018-042535-318161/
├── report.md          what the agent reads
├── report.json        every step, error and timing
├── contact.png        one frame per claim, captioned PASS/FAIL
├── frames/<claim>/NN.png       a screenshot (or terminal view, for `cmd`) after every step
├── frames/<claim>/NN.diff.png  with --compare: what changed since the baseline, in red
├── videos/<claim>.webm
└── app.log            output of the `start` command

Screenshots and video are kept only for claims that FAILed or look different from the baseline, because the agent needs them to fix the code. Passing claims keep their report lines and their frame in contact.png. Only the newest 5 runs are kept. A fully passing run of 12 claims takes about 190 KB instead of 3.2 MB. Use --keep-evidence to keep everything, e.g. to attach a video to a PR.

Visual changes against the base branch

git switch main && provepr run --save-baseline   # screens of every claim → .provepr/baseline/
git switch -    && provepr run --compare         # "looks different from the baseline at step 3" + diff images

Both runs happen on the same machine, so fonts and rendering match and only real changes show. A visual change is reported, not judged: the agent decides whether it was intended.

CI

provepr init writes .github/workflows/provepr.yml. It installs Chromium, runs provepr check --base against the PR's base branch (so a PR can't quietly delete or loosen claims), runs every claim, and uploads .provepr/runs/ as a build artifact even when the job fails. The start: command in the plan boots your app on the CI runner, so add the steps that install your app's dependencies where the workflow's comment shows. Don't hand-write this file: init produces the same tested template everywhere.

What it caught

examples/demo_app has a planted bug: the save handler reads the wrong JSON key, so every save returns a 500, and the page shows "Saved" anyway. We gave Claude Code (headless, with this plugin) a fresh copy and only this prompt:

I built a profile page where you can type your name and save it. Before I open a PR, prove the feature works and fix anything that's broken.

It loaded the skill, wrote its own plan, ran it, and got a FAIL with the HTTP 500. It then fixed the server and the UI that said "Saved" on a failed save, and ran again: 1 PASS · 0 FAIL. In its final message it listed the paths it could not check in a browser. That took 14 turns, and provepr made no LLM calls.

We repeated the test with no plugin at all, only the files provepr init writes, and a prompt that never mentions provepr ("make sure it works and fix anything that's broken"). Claude Code read the rules from AGENTS.md and ran provepr docs. It added two claims while keeping the existing one, used an http step to check that bad input gets a 400, fixed both bugs, and finished at 3 PASS · 0 FAIL in 13 turns.

Run the demo yourself:

cd examples/demo_app && uv run --project ../.. provepr run

How it differs from other tools

• Screen recorders for agents capture what the agent did so a human can review it later. provepr checks claims the agent committed to beforehand and returns a pass/fail result the agent acts on.
• Visual regression tools (screenshot diffing) tell you pixels changed, not whether the feature works.
• AI code reviewers read the diff. provepr runs the change.

Limits

• Web apps and command-line tools. Background jobs, email and native GUIs are UNVERIFIED.
• check --base compares checks exactly. It can't tell whether changing a non-check step (e.g. which page a goto opens) made a claim weaker; the plan diff in the PR shows that.
• The screenshot diff needs two local runs (base branch, then yours) and is not part of the CI template, because it boots the app twice.
• Runs locally. Hosted runs on PR preview deployments (a GitHub App) are on the roadmap.

Roadmap

• Local runner, evidence bundle, CLI
• MCP server, skill, Claude Code and Codex plugin
• Bundled agent guide, provepr init (AGENTS.md, CI workflow), provepr check
• Accumulating plans, --only, http steps for API claims
• Network mocking and failure steps in the browser (mock)
• provepr check --base main: flag claims removed or weakened compared to the base branch
• Screenshot diff against the base branch (--save-baseline, --compare)
• Terminal and CLI apps (cmd)
• GitHub App: run on every PR against its preview deployment (Vercel, Netlify) and post a PR comment

Contributing

See CONTRIBUTING.md.

License

MIT © Orvix Labs