verifaied 0.1.0.dev30

Find what's untested in your Python code — locally, no account required

verifaied

Find what's untested in your Python code, so you (or your coding agent) can fix it without waiting for CI.

Two verbs:

• verifaied check — runs entirely on your machine. No account, no API token, no network call. Tells you which functions are untested or only partially covered, and can hand you a ready-to-paste prompt for each one.
• verifaied upload — syncs the same coverage to the verifAIed app, which adds history, branch diffing against CI, a web UI, and AI-written test prompts. Needs a free account.

Install

uv tool install verifaied        # or
pipx install verifaied

verifaied check — local, no account

pytest --cov --cov-branch --cov-report=json
verifaied check

  12 covered  3 partial  2 untested   (17 functions)

   function        file                   missing
●  apply_topup     app/billing.py:41      42-47
●  refund          app/billing.py:60      61-68
◐  _resolve_plan   app/plans.py:12        18

Nothing leaves your machine — the whole analysis is a local AST pass over the source coverage.json already points at.

Add --prompt to get a concrete, ready-to-paste instruction for each one, generated from the function's signature and its uncovered lines (no LLM, no cost, unlimited):

verifaied check --prompt

Write a pytest test for `apply_topup` in `app/billing.py`.

This function is currently **untested** — no line of it executes under the
existing suite.

What to do:
- Call `apply_topup(user_id, amount)`.
- Assert on the **return value** — compare it to the exact expected value.
- Cover the lines that never execute: **42-47**.
- It raises `ValueError` — cover that path with `pytest.raises(ValueError)`.

Useful flags: --limit N (how many to list, 0 for all), --root <path> (the directory the coverage paths are relative to — pytest's rootdir), and --fail-under <pct>, which exits 3 when too few functions are fully covered, so check works as a CI gate.

verifaied upload — sync to the app

Mint an API token from the verifAIed app (Settings → Tokens; a free account is enough). The CLI ships pointing at the hosted API (https://api.verifaied.app); point it elsewhere via VERIFAIED_API_URL if you're running the backend locally or self-hosting:

export VERIFAIED_API_TOKEN=vr_live_...
# Only set this if you're not using the hosted API:
export VERIFAIED_API_URL=http://localhost:8000   # local dev

From inside any git repo you've connected to verifAIed:

pytest --cov --cov-report=json --junitxml=junit.xml
verifaied upload

That's it — --repo is optional. The CLI parses git remote get-url origin, looks the repo up under your account via /installations/me, and uses the resulting UUID. You can also be explicit:

verifaied upload --repo kyle/verifaied        # owner/name slug
verifaied upload --repo <UUID>                # raw UUID, no lookup

The CLI reads coverage.json, pulls the source for every file it references from your working tree, and posts everything to /repositories/<id>/local-coverage. The response prints a summary of untested / partial / failing functions so you (or your LLM) can fix them on the next iteration.

What gets uploaded

The CLI only sends files that appear in coverage.json's "files" map — i.e. exactly the files pytest --cov instrumented. There is no directory walk and no glob:

• .env, build artefacts, vendored libraries, and anything outside your coverage scope are never read.
• Test files themselves are only uploaded if your coverage config includes them (e.g. --cov=tests).

To audit the exact file list (and total bytes) before anything leaves your machine, run with --dry-run:

verifaied upload --dry-run

This prints the resolved branch / commit / file table and exits without contacting the backend. Useful when you're about to upload from an unfamiliar repo, or when narrowing down where an unexpected file is coming from.

If a sensitive value did make it into a covered source file (a baked-in API key in a fixture, etc.), open the repo in the verifAIed web UI, expand Recently deleted under the branches grid, and use Permanently delete. That hard-deletes the analysis row and cascades to the uploaded source — the per-card Delete is a soft-delete that keeps the data around for accidental-deletion recovery.

upload flags

• --repo / -r <UUID|owner/name> — repository to upload to. If omitted, the CLI auto-detects from the GitHub origin remote.
• --branch / -b <name> — branch to attach the upload to (default: git branch --show-current, then local)
• --coverage <path> — path to coverage.json (default: ./coverage.json)
• --junit <path> — optional JUnit XML for failing-test detail
• --commit-sha <sha> — commit sha for display (default: git rev-parse HEAD)
• --root <path> — root the coverage paths are relative to (default: cwd)
• --api-url <url> — backend base URL (overrides VERIFAIED_API_URL)
• --token <token> — API token (overrides VERIFAIED_API_TOKEN)
• --dry-run — print the file list and total bytes that would be uploaded, then exit without contacting the backend. Skips the token requirement so you can audit without configuring auth.

verifaied check-done — the "am I done?" gate

check-done is the stop condition for an agent's test-writing loop. It fetches a verdict for a branch — failing tests plus the coverage of the functions you changed — and exits accordingly, so a CI job or an agent can gate on it:

pytest --cov --cov-report=json --junitxml=junit.xml
verifaied upload            # push the fresh coverage first
verifaied check-done        # then ask: am I done?

The verdict composes into {done, status, gaps}:

• done (bool) — true only when there are no gaps at all. The strict signal to stop.
• status — done | done-with-warnings | not-done. done-with-warnings means the only gaps left are ones the repo owner marked non-blocking (per-repo policy, set in the web UI).
• gaps[] — each remaining problem with a ref, whether it's blocking, and a next_action pointing at the free (get_baseline_prompt) or paid (fix_branch) remedy.
• not_checked[] — signals that couldn't be evaluated (e.g. failing tests when no junit.xml was uploaded), surfaced so a green verdict is never silently green.

Exit codes

• 0 — done (or done-with-warnings with --allow-warnings)
• 3 — not done: blocking gaps remain (the CI-gate failure code)
• 2 — HTTP/API error (auth, 404, network)
• 1 — usage/config error (no token, bad --repo)

check-done flags

• --repo / -r <UUID|owner/name> — repository to check (default: auto-detect from the GitHub origin remote)
• --branch / -b <name> — branch to judge (default: git branch --show-current)
• --json — print the raw verdict JSON instead of the human summary
• --allow-warnings — treat done-with-warnings as passing (exit 0); default is strict (only a clean done exits 0)
• --api-url <url> — backend base URL (overrides VERIFAIED_API_URL)
• --token <token> — API token (overrides VERIFAIED_API_TOKEN)

As an agent stop condition

Drop this loop into your agent's instructions file so it self-corrects and knows when to stop:

Test-coverage loop (run until done):
  1. Run tests with coverage:  pytest --cov --cov-report=json --junitxml=junit.xml
  2. Push the results:         verifaied upload
  3. Ask if you're done:       verifaied check-done
  4. If exit code is 0, stop — you're done.
     If exit code is 3, fix each gap in the output (use get_baseline_prompt
     for a free test prompt, or fix_branch for an LLM-written one), then
     go back to step 1.

Agent loop

We've found the most fruitful way to use verifAIed is to put a short loop into your coding agent's instructions file (CLAUDE.md, AGENTS.md, .cursorrules, GEMINI.md, .github/copilot-instructions.md) so it self-corrects on every change.

The easiest way is to let your agent write that loop into the rules file for you. Paste the prompt below into your agent — it will inspect the repo, find the project's real test command, confirm the CLI is installed and VERIFAIED_API_TOKEN is set, and write the loop section into the matching rules file, tailored to your repo:

We've found that adding a short test-coverage loop to your agent's instructions is the most fruitful way to use verifAIed. I want you to write that loop into this repo's agent-instructions file, tailored to how this repo actually runs its tests.

verifAIed is a coverage analysis tool. Its CLI (`verifaied upload`) pushes a local `coverage.json` + `junit.xml` to the server; its MCP server exposes a `check_done` tool that returns a `{done, status, gaps}` verdict for the branch (the loop's stop condition), and a `fix_branch` tool that returns a single prompt describing every untested function, partial branch, and failing test. The agent loop is: run tests → `verifaied upload` → call `check_done` → if `done` is true stop, otherwise fix the gaps (`get_baseline_prompt` per function, or `fix_branch` for one prompt covering everything) → repeat.

Do the following, in order:

# 1. Detect the project setup

- **Test runner & command**: look at `pyproject.toml` (`[tool.pytest.ini_options]`, `[project.scripts]`), `pytest.ini`, `tox.ini`, `setup.cfg`, `Makefile` targets (`test`, `check`), `justfile`, or a top-level `scripts/` directory. Use the test command the project already uses — don't invent a new one.
- **Coverage flags**: pytest must produce `coverage.json` with branch coverage AND per-test contexts, plus `junit.xml`. If the existing command already does that, reuse it. Otherwise build the command:

pytest --cov --cov-branch --cov-context=test --junitxml=junit.xml coverage json --show-contexts -o coverage.json

The `coverage json --show-contexts` step is non-negotiable — pytest-cov's `--cov-report=json` alone drops the per-test contexts, which makes verifAIed mark the upload "Needs attention".
- **Agent-instructions file**: pick the one that matches the agent you (the model) are. If unsure, fall back to the project's existing convention.
- Claude Code: `CLAUDE.md`
- Codex CLI / OpenAI: `AGENTS.md`
- Cursor: `.cursorrules` or `.cursor/rules/*.mdc`
- Gemini CLI: `GEMINI.md`
- GitHub Copilot: `.github/copilot-instructions.md`
- If none exists in the repo, create the one matching your own agent.

# 2. Verify the CLI is installed and configured

- Run `verifaied --help`. If it exits non-zero, install it: `uv tool install verifaied` (preferred) or `pipx install verifaied`.
- Check `echo $VERIFAIED_API_TOKEN`. If empty, stop and tell the user to mint one from the verifAIed app (Settings → Tokens) and export it: `export VERIFAIED_API_TOKEN=vr_live_...`. Do NOT invent a placeholder and do NOT proceed without it.

# 3. Write the loop into the agent-instructions file

Append (or create) a section titled `## Test coverage loop with verifAIed` containing:

- The exact test command(s) you built in step 1 — so future runs of the agent don't have to re-derive them.
- The exact upload command (`verifaied upload`, plus any flags this repo needs — e.g. `--coverage path/to/coverage.json` if your test command lands the JSON outside the repo root).
- An instruction to call the `check_done` tool on the verifAIed MCP server after each upload, and to stop only when it returns `done: true`.
- An instruction that while `check_done` reports gaps, fix them: call `get_baseline_prompt(function_id)` for a free per-function test prompt, or `fix_branch` with `scope: everything` for a single prompt covering every gap, then apply it.
- An instruction to repeat from step 1 until `check_done` reports `done: true`.
- One line: "If `VERIFAIED_API_TOKEN` is unset, stop and ask the user — do not skip this loop."

# 4. Confirm with the user

Show the user the diff of the file you edited (or created) and the test command you wrote in. Do NOT run the loop yourself yet — let the user review first.

# Hard rules

- Do NOT change the test command's behavior beyond adding coverage / junit flags.
- Do NOT add new dependencies if pytest-cov and coverage are already transitive deps.
- Do NOT write a fallback that skips the loop when the token is missing — the agent must stop and ask.
- Do NOT enumerate functions yourself or write tests; `check_done` lists the gaps and `fix_branch` / `get_baseline_prompt` return the prompts that close them.
- Do NOT decide "done" yourself; `check_done` is the stop condition.
- Do NOT replace an existing agent-instructions file — append a new section.

Or paste the instructions in by hand

If you'd rather skip the meta-prompt, drop this section into your agent's rules file as-is. Swap in your repo's actual test command if it differs from the pytest default:

## Test coverage loop with verifAIed

After implementing or modifying any code in this repo, run this loop until verifAIed reports done:

1. Run the test suite with branch coverage and per-test contexts:

pytest --cov --cov-branch --cov-context=test --junitxml=junit.xml coverage json --show-contexts -o coverage.json

2. Upload the results to verifAIed:

verifaied upload

3. Ask if you're done: call the `check_done` tool on the verifAIed MCP server (or run `verifaied check-done`). It returns `{done, status, gaps}` for this branch — failing tests plus untested/partially-covered functions among the ones you changed.
4. If `done` is true, stop — the branch is covered.
5. Otherwise fix the gaps it lists: call `get_baseline_prompt(function_id)` for a free test prompt per function, or `fix_branch` with `scope: everything` for a single LLM-written prompt covering every gap on the branch.
6. Go back to step 1.

Stop only when `check_done` reports `done: true`. If `VERIFAIED_API_TOKEN` is not set, stop and ask the user — do not skip this loop.