verifaied

Find what's untested in your code, so you (or your coding agent) can fix it without waiting for CI.

Two verbs:

• verifaied check — runs entirely on your machine. No account, no API token, no network call. Tells you which Python functions are untested or only partially covered, and can hand you a ready-to-paste prompt for each one.
• verifaied upload — syncs the same coverage to the verifAIed app, which adds history, branch diffing against CI, a web UI, and AI-written test prompts. Handles both Python (pytest-cov) and TypeScript/JavaScript (vitest/Istanbul). Needs a free account.

Install

uv tool install verifaied        # or
pipx install verifaied

verifaied check — local, no account

pytest --cov --cov-branch --cov-report=json
verifaied check

  12 covered  3 partial  2 untested   (17 functions)

   function        file                   missing
●  apply_topup     app/billing.py:41      42-47
●  refund          app/billing.py:60      61-68
◐  _resolve_plan   app/plans.py:12        18

Nothing leaves your machine — the whole analysis is a local AST pass over the source coverage.json already points at.

check is Python-only: it reads pytest-cov's coverage.json. Point it at a vitest/Istanbul report and it exits with a note to use verifaied upload, which analyzes TypeScript/JavaScript on the hosted side.

Add --prompt to get a concrete, ready-to-paste instruction for each one, generated from the function's signature and its uncovered lines (no LLM, no cost, unlimited):

verifaied check --prompt

Write a pytest test for `apply_topup` in `app/billing.py`.

This function is currently **untested** — no line of it executes under the
existing suite.

What to do:
- Call `apply_topup(user_id, amount)`.
- Assert on the **return value** — compare it to the exact expected value.
- Cover the lines that never execute: **42-47**.
- It raises `ValueError` — cover that path with `pytest.raises(ValueError)`.

Useful flags: --limit N (how many to list, 0 for all), --root <path> (the directory the coverage paths are relative to — pytest's rootdir), and --fail-under <pct>, which exits 3 when too few functions are fully covered, so check works as a CI gate.

verifaied upload — sync to the app

Mint an API token from the verifAIed app (Settings → Tokens; a free account is enough). The CLI ships pointing at the hosted API (https://api.verifaied.app); point it elsewhere via VERIFAIED_API_URL if you're running the backend locally or self-hosting:

export VERIFAIED_API_TOKEN=vr_live_...
# Only set this if you're not using the hosted API:
export VERIFAIED_API_URL=http://localhost:8000   # local dev

From inside any git repo you've connected to verifAIed:

pytest --cov --cov-report=json --junitxml=junit.xml
verifaied upload

That's it — --repo is optional. The CLI parses git remote get-url origin, looks the repo up under your account via /installations/me, and uses the resulting UUID. You can also be explicit:

verifaied upload --repo kyle/verifaied        # owner/name slug
verifaied upload --repo <UUID>                # raw UUID, no lookup

The CLI reads your coverage report(s), pulls the source for every file they reference from your working tree, and posts everything to /repositories/<id>/local-coverage. With no --coverage flag it auto-discovers the report, looking in order for ./coverage.json (pytest-cov), then ./coverage/coverage-final.json, then a bare ./coverage-final.json (vitest/Istanbul). Pass --coverage one or more times to name reports explicitly — repeat it to push Python and TypeScript coverage in a single upload; multiple reports merge per language on the server. The response prints a summary of untested / partial / failing functions so you (or your LLM) can fix them on the next iteration.

What the upload reports

The summary leads with counts so an LLM (or you) sees the deltas first:

• functions matched — functions in the uploaded report the analyzer resolved against your source.
• functions preserved — functions retained from a previous upload in another language for the same branch (only shown when non-zero). A Python-only upload never wipes the TypeScript functions a vitest upload recorded on the same branch, and vice versa — the branch's total tracked functions is matched + preserved.
• untested / partial / failing tests — what to fix next.

What gets uploaded

The CLI only sends files that appear in your coverage report — i.e. exactly the files your test run instrumented (pytest-cov's "files" map, or the top-level file keys in an Istanbul coverage-final.json). There is no directory walk and no glob:

• .env, build artefacts, vendored libraries, and anything outside your coverage scope are never read.
• Test files themselves are only uploaded if your coverage config includes them (e.g. --cov=tests).

To audit the exact file list (and total bytes) before anything leaves your machine, run with --dry-run:

verifaied upload --dry-run

This prints the resolved branch / commit / file table and exits without contacting the backend. Useful when you're about to upload from an unfamiliar repo, or when narrowing down where an unexpected file is coming from.

If a sensitive value did make it into a covered source file (a baked-in API key in a fixture, etc.), open the repo in the verifAIed web UI, expand Recently deleted under the branches grid, and use Permanently delete. That hard-deletes the analysis row and cascades to the uploaded source — the per-card Delete is a soft-delete that keeps the data around for accidental-deletion recovery.

TypeScript / JavaScript

upload ingests vitest/Istanbul coverage the same way it ingests pytest-cov's. Generate it with vitest's Istanbul provider:

npm i -D @vitest/coverage-istanbul
vitest run --coverage --coverage.provider=istanbul --coverage.reporter=json \
  --reporter=junit --outputFile.junit=junit.xml
verifaied upload --junit junit.xml

The Istanbul provider is required. Istanbul applies your source maps, so coverage/coverage-final.json reports the original .ts / .tsx sources — which is what verifAIed matches function definitions against. vitest's default v8 provider emits a different report shape that verifAIed rejects, so --coverage.provider=istanbul is not optional. The --reporter=junit --outputFile.junit=junit.xml flags produce the JUnit XML that feeds the failing-tests panel; hand it to the upload with --junit.

If a report references only compiled output — paths under dist/, build/, out/, or .next/ with no .ts / .tsx / .jsx entries — the upload is refused. That's the source-maps-not-applied case: coverage landed on bundled JS instead of your sources, and verifAIed can't map it back. Switching to the Istanbul provider is the fix.

Supported formats. pytest-cov coverage.json and Istanbul coverage-final.json only. LCOV, Cobertura XML, jest's own format, and c8 / v8-style JSON are rejected with a clear error naming the two supported shapes.

upload flags

• --repo / -r <UUID|owner/name> — repository to upload to. If omitted, the CLI auto-detects from the GitHub origin remote.
• --branch / -b <name> — branch to attach the upload to (default: git branch --show-current, then local)
• --coverage <path> — coverage report to upload. Repeatable — pass it once per report to combine Python and TypeScript coverage in one upload. Accepts pytest-cov coverage.json and Istanbul coverage-final.json. Default: auto-detect ./coverage.json, then ./coverage/coverage-final.json (or a bare ./coverage-final.json).
• --junit <path> — optional JUnit XML for failing-test detail
• --commit-sha <sha> — commit sha for display (default: git rev-parse HEAD)
• --root <path> — root the coverage paths are relative to (default: cwd)
• --api-url <url> — backend base URL (overrides VERIFAIED_API_URL)
• --token <token> — API token (overrides VERIFAIED_API_TOKEN)
• --dry-run — print the file list and total bytes that would be uploaded, then exit without contacting the backend. Skips the token requirement so you can audit without configuring auth.

verifaied check-done — the "am I done?" gate

check-done is the stop condition for an agent's test-writing loop. It fetches a verdict for a branch — failing tests plus the coverage of the functions you changed — and exits accordingly, so a CI job or an agent can gate on it:

pytest --cov --cov-report=json --junitxml=junit.xml
verifaied upload            # push the fresh coverage first
verifaied check-done        # then ask: am I done?

The verdict composes into {done, status, gaps}:

• done (bool) — true only when there are no gaps at all. The strict signal to stop.
• status — done | done-with-warnings | not-done. done-with-warnings means the only gaps left are ones the repo owner marked non-blocking (per-repo policy, set in the web UI).
• gaps[] — each remaining problem with a ref, whether it's blocking, and a next_action pointing at the free (get_baseline_prompt) or paid (fix_branch) remedy.
• not_checked[] — signals that couldn't be evaluated (e.g. failing tests when no junit.xml was uploaded), surfaced so a green verdict is never silently green.

Exit codes

• 0 — done (or done-with-warnings with --allow-warnings)
• 3 — not done: blocking gaps remain (the CI-gate failure code)
• 2 — HTTP/API error (auth, 404, network)
• 1 — usage/config error (no token, bad --repo)

check-done flags

• --repo / -r <UUID|owner/name> — repository to check (default: auto-detect from the GitHub origin remote)
• --branch / -b <name> — branch to judge (default: git branch --show-current)
• --json — print the raw verdict JSON instead of the human summary
• --allow-warnings — treat done-with-warnings as passing (exit 0); default is strict (only a clean done exits 0)
• --api-url <url> — backend base URL (overrides VERIFAIED_API_URL)
• --token <token> — API token (overrides VERIFAIED_API_TOKEN)

As an agent stop condition

Drop this loop into your agent's instructions file so it self-corrects and knows when to stop:

Test-coverage loop (run until done):
  1. Run tests with coverage:  pytest --cov --cov-report=json --junitxml=junit.xml
  2. Push the results:         verifaied upload
  3. Ask if you're done:       verifaied check-done
  4. If exit code is 0, stop — you're done.
     If exit code is 3, fix each gap in the output (use get_baseline_prompt
     for a free test prompt, or fix_branch for an LLM-written one), then
     go back to step 1.

Agent loop

We've found the most fruitful way to use verifAIed is to put a short loop into your coding agent's instructions file (CLAUDE.md, AGENTS.md, .cursorrules, GEMINI.md, .github/copilot-instructions.md) so it self-corrects on every change.

The easiest way is to let your agent write that loop into the rules file for you. Paste the prompt below into your agent — it will inspect the repo, find the project's real test command, confirm the CLI is installed and VERIFAIED_API_TOKEN is set, and write the loop section into the matching rules file, tailored to your repo:

We've found that adding a short test-coverage loop to your agent's instructions is the most fruitful way to use verifAIed. I want you to write that loop into this repo's agent-instructions file, tailored to how this repo actually runs its tests.

verifAIed is a coverage analysis tool. Its CLI (`verifaied upload`) pushes a local `coverage.json` + `junit.xml` to the server; its MCP server exposes a `check_done` tool that returns a `{done, status, gaps}` verdict for the branch (the loop's stop condition), and a `fix_branch` tool that returns a single prompt describing every untested function, partial branch, and failing test. The agent loop is: run tests → `verifaied upload` → call `check_done` → if `done` is true stop, otherwise fix the gaps (`get_baseline_prompt` per function, or `fix_branch` for one prompt covering everything) → repeat.

Do the following, in order:

# 1. Detect the project setup

- **Test runner & command**: look at `pyproject.toml` (`[tool.pytest.ini_options]`, `[project.scripts]`), `pytest.ini`, `tox.ini`, `setup.cfg`, `Makefile` targets (`test`, `check`), `justfile`, or a top-level `scripts/` directory. Use the test command the project already uses — don't invent a new one.
- **Coverage flags**: pytest must produce `coverage.json` with branch coverage AND per-test contexts, plus `junit.xml`. If the existing command already does that, reuse it. Otherwise build the command:

pytest --cov --cov-branch --cov-context=test --junitxml=junit.xml coverage json --show-contexts -o coverage.json

The `coverage json --show-contexts` step is non-negotiable — pytest-cov's `--cov-report=json` alone drops the per-test contexts, which makes verifAIed mark the upload "Needs attention".
- **Agent-instructions file**: pick the one that matches the agent you (the model) are. If unsure, fall back to the project's existing convention.
- Claude Code: `CLAUDE.md`
- Codex CLI / OpenAI: `AGENTS.md`
- Cursor: `.cursorrules` or `.cursor/rules/*.mdc`
- Gemini CLI: `GEMINI.md`
- GitHub Copilot: `.github/copilot-instructions.md`
- If none exists in the repo, create the one matching your own agent.

# 2. Verify the CLI is installed and configured

- Run `verifaied --help`. If it exits non-zero, install it: `uv tool install verifaied` (preferred) or `pipx install verifaied`.
- Check `echo $VERIFAIED_API_TOKEN`. If empty, stop and tell the user to mint one from the verifAIed app (Settings → Tokens) and export it: `export VERIFAIED_API_TOKEN=vr_live_...`. Do NOT invent a placeholder and do NOT proceed without it.

# 3. Write the loop into the agent-instructions file

Append (or create) a section titled `## Test coverage loop with verifAIed` containing:

- The exact test command(s) you built in step 1 — so future runs of the agent don't have to re-derive them.
- The exact upload command (`verifaied upload`, plus any flags this repo needs — e.g. `--coverage path/to/coverage.json` if your test command lands the JSON outside the repo root).
- An instruction to call the `check_done` tool on the verifAIed MCP server after each upload, and to stop only when it returns `done: true`.
- An instruction that while `check_done` reports gaps, fix them: call `get_baseline_prompt(function_id)` for a free per-function test prompt, or `fix_branch` with `scope: everything` for a single prompt covering every gap, then apply it.
- An instruction to repeat from step 1 until `check_done` reports `done: true`.
- One line: "If `VERIFAIED_API_TOKEN` is unset, stop and ask the user — do not skip this loop."

# 4. Confirm with the user

Show the user the diff of the file you edited (or created) and the test command you wrote in. Do NOT run the loop yourself yet — let the user review first.

# Hard rules

- Do NOT change the test command's behavior beyond adding coverage / junit flags.
- Do NOT add new dependencies if pytest-cov and coverage are already transitive deps.
- Do NOT write a fallback that skips the loop when the token is missing — the agent must stop and ask.
- Do NOT enumerate functions yourself or write tests; `check_done` lists the gaps and `fix_branch` / `get_baseline_prompt` return the prompts that close them.
- Do NOT decide "done" yourself; `check_done` is the stop condition.
- Do NOT replace an existing agent-instructions file — append a new section.

Or paste the instructions in by hand

If you'd rather skip the meta-prompt, drop this section into your agent's rules file as-is. Swap in your repo's actual test command if it differs from the pytest default:

## Test coverage loop with verifAIed

After implementing or modifying any code in this repo, run this loop until verifAIed reports done:

1. Run the test suite with branch coverage and per-test contexts:

pytest --cov --cov-branch --cov-context=test --junitxml=junit.xml coverage json --show-contexts -o coverage.json

2. Upload the results to verifAIed:

verifaied upload

3. Ask if you're done: call the `check_done` tool on the verifAIed MCP server (or run `verifaied check-done`). It returns `{done, status, gaps}` for this branch — failing tests plus untested/partially-covered functions among the ones you changed.
4. If `done` is true, stop — the branch is covered.
5. Otherwise fix the gaps it lists: call `get_baseline_prompt(function_id)` for a free test prompt per function, or `fix_branch` with `scope: everything` for a single LLM-written prompt covering every gap on the branch.
6. Go back to step 1.

Stop only when `check_done` reports `done: true`. If `VERIFAIED_API_TOKEN` is not set, stop and ask the user — do not skip this loop.