verifaied

Find what's untested in your code, so you (or your coding agent) can fix it without waiting for CI.

Two verbs:

• verifaied check — runs entirely on your machine. No account, no API token, no network call. Tells you which Python functions are untested or only partially covered, and can hand you a ready-to-paste prompt for each one.
• verifaied upload — syncs the same coverage to the verifAIed app, which adds history, branch diffing against CI, a web UI, and AI-written test prompts. Handles both Python (pytest-cov) and TypeScript/JavaScript (vitest/Istanbul). Needs a free account.

Install

uv tool install verifaied        # or
pipx install verifaied

verifaied check — local, no account

pytest --cov --cov-branch --cov-report=json
verifaied check

  12 covered  3 partial  2 untested   (17 functions)

   function        file                   missing
●  apply_topup     app/billing.py:41      42-47
●  refund          app/billing.py:60      61-68
◐  _resolve_plan   app/plans.py:12        18

Nothing leaves your machine — the whole analysis is a local AST pass over the source coverage.json already points at.

check is Python-only: it reads pytest-cov's coverage.json. Point it at a vitest/Istanbul report and it exits with a note to use verifaied upload, which analyzes TypeScript/JavaScript on the hosted side.

Add --prompt to get a concrete, ready-to-paste instruction for each one, generated from the function's signature and its uncovered lines (no LLM, no cost, unlimited):

verifaied check --prompt

Write a pytest test for `apply_topup` in `app/billing.py`.

This function is currently **untested** — no line of it executes under the
existing suite.

What to do:
- Call `apply_topup(user_id, amount)`.
- Assert on the **return value** — compare it to the exact expected value.
- Cover the lines that never execute: **42-47**.
- It raises `ValueError` — cover that path with `pytest.raises(ValueError)`.

Useful flags: --limit N (how many to list, 0 for all), --root <path> (the directory the coverage paths are relative to — pytest's rootdir), and --fail-under <pct>, which exits 3 when too few functions are fully covered, so check works as a CI gate.

verifaied upload — sync to the app

Mint an API token from the verifAIed app (Settings → Tokens; a free account is enough). The CLI ships pointing at the hosted API (https://api.verifaied.app); point it elsewhere via VERIFAIED_API_URL if you're running the backend locally or self-hosting:

export VERIFAIED_API_TOKEN=vr_live_...
# Only set this if you're not using the hosted API:
export VERIFAIED_API_URL=http://localhost:8000   # local dev

From inside any git repo you've connected to verifAIed:

pytest --cov --cov-report=json --junitxml=junit.xml
verifaied upload

That's it — --repo is optional. The CLI parses git remote get-url origin, looks the repo up under your account via /installations/me, and uses the resulting UUID. You can also be explicit:

verifaied upload --repo kyle/verifaied        # owner/name slug
verifaied upload --repo <UUID>                # raw UUID, no lookup

The CLI reads your coverage report(s), pulls the source for every file they reference from your working tree, and posts everything to /repositories/<id>/local-coverage. With no --coverage flag it auto-discovers the report, looking in order for ./coverage.json (pytest-cov), then ./coverage/coverage-final.json, then a bare ./coverage-final.json (vitest/Istanbul). Pass --coverage one or more times to name reports explicitly — repeat it to push Python and TypeScript coverage in a single upload; multiple reports merge per language on the server. The response prints a summary of untested / partial / failing functions so you (or your LLM) can fix them on the next iteration.

What the upload reports

The summary leads with counts so an LLM (or you) sees the deltas first:

• functions matched — functions in the uploaded report the analyzer resolved against your source.
• functions preserved — functions retained from a previous upload in another language for the same branch (only shown when non-zero). A Python-only upload never wipes the TypeScript functions a vitest upload recorded on the same branch, and vice versa — the branch's total tracked functions is matched + preserved.
• untested / partial / failing tests — what to fix next.

What gets uploaded

The CLI only sends files that appear in your coverage report — i.e. exactly the files your test run instrumented (pytest-cov's "files" map, or the top-level file keys in an Istanbul coverage-final.json). There is no directory walk and no glob:

• .env, build artefacts, vendored libraries, and anything outside your coverage scope are never read.
• Test files themselves are only uploaded if your coverage config includes them (e.g. --cov=tests).

To audit the exact file list (and total bytes) before anything leaves your machine, run with --dry-run:

verifaied upload --dry-run

This prints the resolved branch / commit / file table and exits without contacting the backend. Useful when you're about to upload from an unfamiliar repo, or when narrowing down where an unexpected file is coming from.

If a sensitive value did make it into a covered source file (a baked-in API key in a fixture, etc.), open the repo in the verifAIed web UI, expand Recently deleted under the branches grid, and use Permanently delete. That hard-deletes the analysis row and cascades to the uploaded source — the per-card Delete is a soft-delete that keeps the data around for accidental-deletion recovery.

TypeScript / JavaScript

upload ingests vitest/Istanbul coverage the same way it ingests pytest-cov's. Generate it with vitest's Istanbul provider:

npm i -D @vitest/coverage-istanbul
vitest run --coverage --coverage.provider=istanbul --coverage.reporter=json \
  --reporter=junit --outputFile.junit=junit.xml
verifaied upload --junit junit.xml

The Istanbul provider is required. Istanbul applies your source maps, so coverage/coverage-final.json reports the original .ts / .tsx sources — which is what verifAIed matches function definitions against. vitest's default v8 provider emits a different report shape that verifAIed rejects, so --coverage.provider=istanbul is not optional. The --reporter=junit --outputFile.junit=junit.xml flags produce the JUnit XML that feeds the failing-tests panel; hand it to the upload with --junit.

If a report references only compiled output — paths under dist/, build/, out/, or .next/ with no .ts / .tsx / .jsx entries — the upload is refused. That's the source-maps-not-applied case: coverage landed on bundled JS instead of your sources, and verifAIed can't map it back. Switching to the Istanbul provider is the fix.

Supported formats. pytest-cov coverage.json and Istanbul coverage-final.json only. LCOV, Cobertura XML, jest's own format, and c8 / v8-style JSON are rejected with a clear error naming the two supported shapes.

upload flags

• --repo / -r <UUID|owner/name> — repository to upload to. If omitted, the CLI auto-detects from the GitHub origin remote.
• --branch / -b <name> — branch to attach the upload to (default: git branch --show-current, then local)
• --coverage <path> — coverage report to upload. Repeatable — pass it once per report to combine Python and TypeScript coverage in one upload. Accepts pytest-cov coverage.json and Istanbul coverage-final.json. Default: auto-detect ./coverage.json, then ./coverage/coverage-final.json (or a bare ./coverage-final.json).
• --junit <path> — optional JUnit XML for failing-test detail
• --commit-sha <sha> — commit sha for display (default: git rev-parse HEAD)
• --root <path> — root the coverage paths are relative to (default: cwd)
• --api-url <url> — backend base URL (overrides VERIFAIED_API_URL)
• --token <token> — API token (overrides VERIFAIED_API_TOKEN)
• --dry-run — print the file list and total bytes that would be uploaded, then exit without contacting the backend. Skips the token requirement so you can audit without configuring auth.

verifaied check-done — the "am I done?" gate

check-done is the stop condition for an agent's test-writing loop. It fetches a verdict for a branch — failing tests plus the coverage of the functions you changed — and exits accordingly, so a CI job or an agent can gate on it:

pytest --cov --cov-report=json --junitxml=junit.xml
verifaied upload            # push the fresh coverage first
verifaied check-done        # then ask: am I done?

The verdict composes into {done, status, gaps}:

• done (bool) — true only when there are no gaps at all. The strict signal to stop.
• status — done | done-with-warnings | not-done. done-with-warnings means the only gaps left are ones the repo owner marked non-blocking (per-repo policy, set in the web UI).
• gaps[] — each remaining problem with a ref, whether it's blocking, and a next_action pointing at the free (get_baseline_prompt) or paid (fix_branch) remedy.
• not_checked[] — signals that couldn't be evaluated (e.g. failing tests when no junit.xml was uploaded), surfaced so a green verdict is never silently green.

Exit codes

• 0 — done (or done-with-warnings with --allow-warnings)
• 3 — not done: blocking gaps remain (the CI-gate failure code)
• 2 — HTTP/API error (auth, 404, network)
• 1 — usage/config error (no token, bad --repo)

check-done flags

• --repo / -r <UUID|owner/name> — repository to check (default: auto-detect from the GitHub origin remote)
• --branch / -b <name> — branch to judge (default: git branch --show-current)
• --json — print the raw verdict JSON instead of the human summary
• --allow-warnings — treat done-with-warnings as passing (exit 0); default is strict (only a clean done exits 0)
• --api-url <url> — backend base URL (overrides VERIFAIED_API_URL)
• --token <token> — API token (overrides VERIFAIED_API_TOKEN)

As an agent stop condition

Drop this loop into your agent's instructions file so it self-corrects and knows when to stop:

Test-coverage loop (run until done):
  1. Run tests with coverage:  pytest --cov --cov-report=json --junitxml=junit.xml
  2. Push the results:         verifaied upload
  3. Ask if you're done:       verifaied check-done
  4. If exit code is 0, stop — you're done.
     If exit code is 3, fix each gap in the output (use get_baseline_prompt
     for a free test prompt, or fix_branch for an LLM-written one), then
     go back to step 1.

verifaied audit — measure the rendered page

Coverage says a browser reached your component. It can't say the button ended up behind the modal. verifaied audit loads your running app at several viewport widths and measures it — no LLM, no cost:

• Overflow — anything spilling out of the viewport, page or element.
• Contrast — text below the WCAG AA minimum (4.5:1, or 3:1 for large text), skipping anything sitting on a background image, where computed styles can't answer honestly.
• Broken images — an <img> that finished loading with nothing in it.
• Console errors — what the page shouted while it rendered.

It drives a real browser, so Playwright ships as an optional extra rather than a dependency of every command:

pip install 'verifaied[audit]'
playwright install chromium

verifaied audit http://localhost:5173

  viewport   findings   console errors
  375x812           2                0
  768x1024          0                0
  1280x800          0                0

375x812  http://localhost:5173/
  element_overflow  div.card  Extends 40px past the right edge of the 375px viewport.
  low_contrast      p.muted   Text contrast is 2.10:1 against its background, below…

Findings are recorded against the branch as visual_defect gaps — the one check_done signal that is a warning by default. An audit doesn't know which of your choices were deliberate (a horizontally-scrolling table, a muted caption), so it reports loudly and blocks nothing until you turn it on under Definition of done. Only the most recent audit counts: re-run it after a fix and a clean pass clears what the last one raised.

Each viewport's full-page screenshot is stored with the run, so every finding has its frame beside it.

Exit codes

• 0 — clean: nothing found at any viewport
• 3 — findings recorded (the CI-gate failure code)
• 2 — the audit could not be run or could not be recorded
• 1 — usage/config error (no token, bad --repo, bad --viewports, or Playwright not installed)

audit flags

• --repo / -r <UUID|owner/name> — repository to record against (default: auto-detect from the GitHub origin remote)
• --branch / -b <name> — branch (default: git branch --show-current)
• --viewports <widths> — comma-separated widths in pixels (default: 375,768,1280)
• --driver <human|agent|suite> — who's running it, for the session ledger (default: human)
• --api-url <url> — backend base URL (overrides VERIFAIED_API_URL)
• --token <token> — API token (overrides VERIFAIED_API_TOKEN)

verifaied test — record a flow once, replay it forever

A coverage percentage says which lines ran. It can't say whether the API-tokens screen still works. verifaied test records that flow in a real browser, saves it as a normal @playwright/test spec in your repo, and replays it with video whenever you ask.

Name the flow in verifAIed (New test on the branch's Tests page, which opens a page of its own) — or just record it, and the CLI creates the row for you:

# Record. A browser opens; click the flow through, and say what you expect
# from the recording studio in verifAIed. Click Finish recording when done
# (closing the browser by hand does the same thing). Set base_url in
# .verifaied/config.toml and this is just `--url /settings`.
verifaied test record "Settings/API tokens" --url http://localhost:3000/settings

# Start from another recorded flow — it replays first, visibly, and is
# never re-recorded. Repeatable and ordered. It is also somewhere to
# start, so --url is optional: recording begins where the chain ends.
verifaied test record "Settings/API tokens" --pre "Auth/Log in"

# With neither, recording starts at base_url — so a flow that begins on
# your app's home page needs nothing but a name.
verifaied test record "Home/Sign up"

# Change one step without redoing the flow. The first two steps replay,
# then the recorder hands you the browser at exactly that point — with
# the original's remaining steps beside the canvas to copy across.
verifaied test edit "Settings/API tokens" --from 2

# Replay it — headless, with video, trace and browser coverage.
verifaied test run "Settings/API tokens"

# Or replay everything and exit with the worst code.
verifaied test run --all

# Point any of them at another configured environment.
verifaied test run --all --env stage

# What's recorded, and how each last went on this branch.
verifaied test list

# Or leave a recorder running and drive the whole thing from the web.
verifaied test studio

A test is named <feature path>/<name>, parsed at its last slash. The feature path is optional and may nest as deep as you like, so "Log in" is a test at the root of the tree and "Settings/API tokens/Revoke" is one two folders down — which is why the name itself may never contain a slash.

The spec lands at .verifaied/tests/<feature path>/<name>.spec.ts — yours to read, edit and commit. Recording is followed immediately by a replay, because the recorder can't capture video and a spec that doesn't pass isn't a recording: the replay is what produces the video you watch back in verifAIed, the trace, and the browser coverage for the branch.

The recording window is a real window — resize it and the page resizes with it, so you can record a flow at the width you care about. While you click, the steps and assertions appear live on the branch's Tests page in verifAIed, and stay there afterwards as the test's flowchart.

Close the recorder without doing anything and the test stays a draft — nothing is saved, and nothing starts holding your branch to it.

Editing a saved flow is verifaied test edit "<ref>" --from N, and it is a branch rather than a re-record: the first N steps are kept and replayed one at a time — the flow ticks down to the branch point like a debugger — and from there you rebuild with the whole studio. --from 0 starts again from nothing and --from <step count> adds to the end. The kept steps are frozen for the session, so to change step 3 you branch at 2: the branch point is the edit cursor. Whatever came after it is listed beside the canvas in verifAIed and can be copied across a step or a group at a time, each one executed against the live page exactly like a line you write. Finishing overwrites the spec; cancelling leaves it byte for byte as it was.

A test this branch has never run, or whose last run failed, shows up as the unverified_recorded_test gap in verifaied check-done.

Saying what you expect

Assertions are authored in the recording studio — the page the record command puts you on, which comes alive when the recording connects — and not in the page being recorded: Playwright's own in-page toolbar offers four canned matchers and sits on top of the UI you are pointing at, so this recorder switches it off.

Click Pick, then click an element in the recording browser. That click is swallowed rather than recorded, so choosing something never becomes a step of the flow — and the picker is verifAIed's own, so it reaches disabled controls and elements the page has made click-through, which the stock one cannot. Escape cancels it. With an element in hand, the builder composes any expect matcher — visible, hidden, enabled, disabled, checked, text, value, CSS property, attribute, class, count, URL — and there is a paid box beside it that turns "this button should be red" into the same kind of line, written from the element's HTML and computed styles.

Either way, the running page decides. Every assertion is executed against the browser you have open before it is kept: one that holds is written into the spec where you added it and appears in the flowchart with its own accent, and one that doesn't comes back with Playwright's own received-vs-expected text instead of quietly becoming a line that asserts nothing.

Doing something the recorder can't record

Chromium dispatches no events on a disabled control, so clicking one in the recording browser records nothing — which makes "click the disabled Save button, then check nothing changed" a flow you cannot capture by clicking. The studio's Action builder is the way in: pick the element, choose a verb (click, double-click, fill, press, check, uncheck, select an option, hover) and click Add action. The action is performed against the live page and only kept if it ran, exactly like an assertion.

When the picked element is disabled, Force is switched on for you — that's what makes the input land on a control the browser would otherwise skip — and the composed line says so, .click({ force: true }), so the saved spec reads as what it does. Authored actions appear in the flowchart with the same accent as authored assertions, and are edited and deleted the same way.

Describing a whole flow

Describe a flow takes several steps at once, in your own words — "fill my email and password, sign in, and check the dashboard heading shows". verifAIed composes the lines against a snapshot of the page as it stands right now, sent up by the recorder alongside the element you picked, and hands them back as one ordered batch. Paid, and the one place a model writes actions as well as assertions: you wrote the steps, and the running page still gets the last word on every one of them.

The CLI holds the batch and releases it one step at a time — the next line only goes to the browser once the one before it has actually held. The moment a step is refused the sequence stops: everything behind it is cancelled without ever running, and the studio shows the one step that failed with Playwright's own words. So a wrong line costs you a red block and a retry, not a sequence that carried on into whatever it found next.

Changing what the recorder captured

While the recording is live, every block in the flow can be edited or deleted — the assertions and actions the studio authored, and the natural clicks the recorder wrote down as you made them. An edit opens the same builder that composes a new line, prefilled; a rewrite is run against the page exactly like the original and takes the original's place in the flow, so a mis-clicked step is a correction rather than a re-recording.

The one block with no controls is the navigation the spec starts from: a spec without its goto cannot replay at all, so removing one would break the test rather than change it.

Deleting one of the recorder's own lines does not edit the recorder's file — nothing can, it is rewritten in full on every action — so the CLI stops emitting that line when it saves the spec instead. That is why the delete carries the statement's text as well as its position: the recorder mutates lines it has already written (a download or a dialog rewrites an earlier one; a run of keystrokes collapses into a single fill), and if the text no longer matches when the command is applied, the delete is refused and the step comes back rather than taking a neighbour with it.

Once the spec is saved it is a plain file in your repo, and your editor is the better tool for it.

Running another test inside this one

The rail's Insert a test section runs a whole recorded flow at the point you are in this one: pick a test, click Insert, and its steps run against the page in the recording browser. If they run, they join the flow as one named block and are written into the saved spec as a single test.step(...):

await test.step("Run 'Auth/Log in'", async () => {
  await page.goto('http://localhost:3000/login');
  await page.getByLabel('Email').fill('a@b.c');
});

That is ordinary Playwright — the saved spec runs for anybody, with or without verifAIed. Only that test's own steps are spliced in, never its pre-step chain: a chain is set-up built for a fresh browser, and replaying "log in" in the middle of a flow that is already logged in is at best a detour. The same test can be inserted as many times as the flow needs it, because an insert happens at a position.

Keep linked is the per-insert choice, off by default:

• Off (snapshot) — the steps are copied in and now belong to this spec. They never change on their own.
• On (linked) — the block carries a marker comment naming the test it came from and the hash of the body it copied, and verifAIed rewrites it when that test is re-recorded. Until then the Tests page says the copy is out of date, and the next verifaied test run (or the replay after a recording) refreshes it in place and tells you it did. Files are only rewritten when something actually changed.

Finish recording saves the spec and closes the browser, which is the same thing closing the window by hand does.

Pre-steps

--pre "<Feature>/<Name>" names another recorded test to run first. It replays in the browser before recording starts, so signing in is something you do once rather than at the head of every spec. Pass it more than once for a chain; the order is the order they run, a pre-step's own pre-steps run before it, and a flow shared by two of them runs once.

A chain is also somewhere to start: with one in front of it a test needs no --url at all, and recording begins on the page the chain leaves the browser on. A test recorded that way has no navigation of its own, so its chain can't be cleared — verifAIed refuses, and test run says so if the chain is scrubbed by deleting a test it named.

The chain is stored with the test, so later test record and test run calls reuse it without the flag. Each spec file in your repo stays a plain recording of one flow — the composition happens in a throwaway file at run time, which is what makes editing Auth/Log in fix every test that starts by logging in. You can also pick pre-steps on the New test page.

Environments

A recorded flow runs against a real app somewhere, and somewhere is the thing that changes: your machine, a container stack, staging. Name each one, and everything that differs about running there lives in that one table:

# .verifaied/config.toml — commit this alongside your specs.
[browser-tests]
default = "local"

[browser-tests.env.local]
base_url = "http://localhost:3000"
reset = "make db-reset"

[browser-tests.env.stage]
base_url = "https://stage.example.com"
# No reset. A deployed app is not yours to drop.

verifaied test run "Settings/API tokens"            # default → local
verifaied test run "Settings/API tokens" --env stage
verifaied test record "Settings/API tokens" --env stage --url /settings
verifaied test studio --env stage

--env works on record, run and studio. Without it you get default, or the only environment you defined; two or more with nothing choosing between them is an error rather than a guess, and a name that isn't configured is refused with the ones that are.

The flat reset and base_url keys still work and read as one unnamed environment, so a config written before environments existed carries on unchanged.

base_url is what keeps a recording from belonging to one machine. Start a flow at --url /settings and every location the recorder writes down on that origin is saved as a path, so the spec in your repo reads await page.goto('/settings') and replays against whatever base_url says on the machine running it. It becomes Playwright's baseURL for the run, and it is also where a recording starts when nothing else says: no --url and no --pre records from /.

Absolute URLs are left exactly as they are, on both sides: a flow that genuinely starts on somebody else's login page keeps working, and a spec full of absolute URLs replays as it always did. base_url must be an origin — scheme and host, no path — and a spec with a relative location in a repo that names none is refused up front, rather than dying on Playwright's "invalid URL" halfway through a run.

Variables — the data that differs, and the data that must be fresh

A login is one flow with three passwords. A "create a token called scratch" flow works once and then meets the token it made. Both are about data, not about where the app is, so both live in the environment too:

[browser-tests.env.local.vars]
# Committed, non-secret: the same for everyone who checks this out.
login_email = "test@example.com"
# Secret: this machine's environment supplies it.
login_password = { from_env = "E2E_PASSWORD" }
# Fresh every run: "tok-e2e-9f3ab120".
token_name = { value = "tok-e2e", unique = true }

Type a configured value into the form while recording and the spec in your repo reads:

await page.getByLabel('Email').fill(process.env.VERIFAIED_VAR_LOGIN_EMAIL);

The live flowchart shows it as ${login_email}, and every run supplies whatever that environment says. The rewrite is whole-literal and exact: a value that merely appears inside a longer string is somebody's own text and is left alone, press is never touched (its argument is a key, not data), and goto/toHaveURL belong to base_url.

from_env is a secret. It never reaches verifAIed: not in the spec, not in the page snapshot the studio reasons about, not in the text of a failed assertion — every one of those has the value replaced by ${name} on the way out. A missing one stops the run by name, before the reset, because finding out afterwards costs you a database.

unique is what makes a deployed environment work without a reset. The value gets a fresh suffix per run — one value for the whole invocation, so every reference to it agrees — so the flow that creates tok-e2e-9f3ab120 can run again tomorrow against the same database. That is the intended answer for staging and prod, and it is why they need no reset at all.

Names are lowercase ([a-z][a-z0-9_]*), values are at least three characters (a shorter one would match all over a spec), and anything malformed is an error naming the variable rather than a silently ignored line.

verifaied var set — write one without opening the file

# Committed with the file.
verifaied var set login_email "test@example.com"

# A secret: only the environment variable's NAME is written down.
verifaied var set login_password --from-env E2E_PASSWORD

# Fresh every run.
verifaied var set token_name "tok-e2e" --unique

# Which environment. A name that doesn't exist yet is created — this is
# how the first one gets written.
verifaied var set login_email "test@stage.com" --env stage

It writes into the environment a run would use — --env, else the configured default, else the only one defined — and refuses rather than guessing when there are several and nothing chooses between them. A repo with no environments at all is told to pass --env <name>, because a variable belongs to one.

Your comments, key ordering and formatting survive: the file is one your repo commits and reviews, so it is edited rather than regenerated. And the write is verified by reading it back — if the result wouldn't load, the original bytes go back and the command fails.

This is what the studio's Variables section hands you while you are recording: name a variable there and the page shows the exact line to run on the machine with the recorder on it.

A recording picks the change up on its own. The config is re-read on the recorder's own beat — a stat per second, and a parse only when the file actually moved — so running var set with the browser still open adds the name to the studio's list within a couple of seconds and hands the value straight to the running driver, where the next fill resolves it instead of typing undefined. A unique variable keeps the value it started the recording with, and a config that is momentarily unparseable (a half-saved buffer) is skipped in silence rather than taking a live browser down with it.

One thing is not automatic, and cannot be: a from_env value is read out of the environment of the process running the recorder, so exporting it in another shell can never reach it. Export it and restart verifaied test record / verifaied test studio — until then the variable is listed as unset and simply isn't offered, rather than resolving to nothing.

A line that names a variable with no value on this machine is refused before it reaches the browser, with the command that fixes it: ${api_token} has no value on this machine — run: verifaied var set api_token "<value>" — it applies automatically, no restart. Playwright's own answer would be value: expected string, got undefined, which is true and tells nobody what to do.

verifaied var unset — remove one

verifaied var unset login_email

# Which environment. Unlike `var set`, it has to exist already.
verifaied var unset login_email --env stage

The same environment precedence, the same comment-preserving edit, and the same verify-by-reload: if the result wouldn't load, or the variable still reads back, the original bytes go back and nothing changed. A name the environment doesn't declare is refused with the ones it does. The vars table is left in place even when it ends up empty — the header and the comment above it are yours.

This is what the studio's Variables list hands over for its Trash button: the config is a file on your machine, so the command is the delete.

Resetting state before each run

A recorded flow runs against a real application with real data. "Create a token called scratch" passes the first time and fails forever after, because the second run meets the token the first one made. Tell verifAIed how to put your app back, in the environment where that is safe to do:

[browser-tests.env.local]
base_url = "http://localhost:3000"
reset = "make db-reset"

Commit that file — it belongs to the repo, the same way the specs do, and everyone who runs the tests should be running the same reset.

The command is a shell line, run from --root (your repo root by default) with your terminal's stdio, so you watch it work. There is no timeout: a database reset can legitimately take minutes, and Ctrl-C still reaches it. It runs:

• before every recording, chain or no chain — what a flow is recorded against has to be what it is later replayed against;
• before the replay that follows a recording, because the recording just spent a few minutes changing everything it touched;
• before each spec in verifaied test run, so every flow meets the same starting state.

It is fail-closed. A reset that exits non-zero stops the run before any session is opened — no video, no coverage, no pass recorded — and a [browser-tests] section the CLI cannot read (malformed TOML, a reset that isn't a non-empty string) is an error too, because a typo must never read as "no reset wanted". No reset key means no reset; a named environment without one says so in one dim line, so you are never left wondering whether the config was read.

A reset aimed somewhere that isn't your machine is refused. An environment with a reset and a non-loopback base_url is an error at load unless it also says reset_remote = true. Deployed environments are not meant to need one — use unique variables instead — and the opt-in exists for the one real case, a dedicated throwaway e2e deployment, where it has to be said out loud.

The config is read fresh at each of those points, so editing it takes effect on the next recording without restarting test studio.

The command lives in your repo rather than in verifAIed because it is a shell line executed on developer machines: that is exactly the trust model of an npm script or a git hook, reviewed by the people who work on the repo. Storing it server-side would turn a compromised account into code running on every laptop that pulled.

Where a run happened

Every run says which environment produced it: beside the run in the ledger, on the Tests page's last-run line, and on the connected recorder's pill (Recorder connected · main · runs against stage). It is a label, not a verdict — check_done asks whether a recorded test passed, not where — so a pass on stage clears the gap exactly as a pass locally does.

Tests tracked from your own Playwright suite are outside this: their playwright.config.ts owns where they point, and verifAIed only reads the report they produced.

test studio — record from the web, without the terminal

verifaied test record is one recording per terminal command. verifaied test studio is the same recording with the terminal taken out of the loop: start it once, leave it running, and from then on the whole flow — name the test, give it a start URL or a chain to start from, click Record — happens on the verifAIed Tests page. The browser still opens on your machine, because that is where your app is.

verifaied test studio

While it is connected the Tests page shows a Recorder connected pill with the branch it is on, and every test gets a Record button in place of the command to paste. The Playwright version gate runs before any of that is announced, so a machine that could never record never claims it can.

Running from the web. A connected studio does two jobs, not one. The Run button on a recorded test's page (repos/:repoId/tests/:testId/view) queues a run on the same queue a Record queues a recording on, and the studio replays that test's saved spec headed — you clicked it, you are sitting in front of this machine, so watching the browser do it is the point. verifaied test run from a terminal stays headless. Either way the replay streams its progress back as it goes, and the flowchart on the page fills in block by block: a tick on what has passed, a pulse on what is running, a cross where it stopped. The verdict that lands when the run finishes is the same one verifaied test run produces — the streaming is what somebody watches, not what is recorded.

A studio older than 0.15.0 cannot run, only record. It says so on every beat, the service only hands it what it declared, and clicking Run against one is refused with "the connected studio is too old — restart it after upgrading verifaied" rather than queueing something that would wait forever.

Queueing. Recordings and runs happen one at a time, oldest first — one studio can drive one browser, and a spec being re-recorded must not be replayed at the same time. Clicking Record on a second test while the first is going queues it, and the web says what it is waiting behind. A queued recording can be cancelled from the studio page until a recorder takes it; after that there is a browser opening on your machine, and nothing in the cloud can stop it. A request that fails is reported with the reason and the studio carries on — one bad pre-step chain is not a reason for it to die.

Branches. The branch is re-detected on every beat, so switching branches under a running studio moves what the web shows and what the next recording lands against. --branch pins it instead.

Ctrl-C does the safe thing at either moment. Mid-recording, the interrupt is forwarded to the recorder exactly as it is for test record: the browser closes, the spec is saved, the replay runs, the request completes — and only then does the studio exit. At the idle prompt it stops immediately and says goodbye, so the connected pill disappears at once rather than timing out.

Exit codes: 0 stopped cleanly, 1 usage/config error (no token, bad --repo, unsupported Playwright), 2 the API could not be reached. A dropped heartbeat is warned about and retried; a recording the studio could not report on is not something to keep quiet about, so that one stops it.

Requirements

node, npx and @playwright/test in your repo (or a subdirectory of it — the CLI finds the first one that has it, and --runner-dir overrides). This is Node Playwright, not the Python verifaied[audit] extra.

test record and test studio only run against Playwright 1.60.x. Recording drives Playwright's recorder through an internal API — the same entry point playwright codegen uses — so it refuses any version it hasn't been verified against rather than opening a browser that might record nothing. test run uses only public API and works on any version.

Exit codes

• 0 — recorded, or replayed cleanly
• * — the replay's own exit code when the spec failed (test run --all reports the worst across the run)
• 2 — the replay was clean but its final coverage upload failed
• 1 — usage/config error (no token, bad --repo, unknown test, no Playwright)

test flags

Shared by record, run and studio:

• --repo / -r <UUID|owner/name> — repository (default: auto-detect from the GitHub origin remote)
• --branch / -b <name> — branch (default: git branch --show-current)
• --commit <sha> — commit SHA (default: git rev-parse HEAD)
• --root <dir> — repo root the spec paths resolve against (default: cwd)
• --runner-dir <dir> — where to run npx playwright from (default: the first directory under --root with @playwright/test installed)
• --port <n> — coverage collector port, 127.0.0.1 only (default: 4571)
• --interval <seconds> — seconds between coverage uploads during a replay (default: 10)
• --env <name> — which environment from .verifaied/config.toml to run against (default: the configured default, or the only one defined)
• --api-url <url> / --token <token>

record also takes --url <path-or-url> (optional: a path is resolved against base_url, and with no --url recording starts where a --pre chain leaves the browser, or at base_url itself when there is no chain either), --description <text>, --pre "<Feature>/<Name>" (repeatable; replaces the stored chain — omit it to keep what the test already has), and --no-replay to skip the replay. run takes --all in place of a test name. studio takes --no-replay and takes no test name at all — what it records is whatever the web asks for.

Agent loop

We've found the most fruitful way to use verifAIed is to put a short loop into your coding agent's instructions file (CLAUDE.md, AGENTS.md, .cursorrules, GEMINI.md, .github/copilot-instructions.md) so it self-corrects on every change.

The easiest way is to let your agent write that loop into the rules file for you. Paste the prompt below into your agent — it will inspect the repo, find the project's real test command, confirm the CLI is installed and VERIFAIED_API_TOKEN is set, and write the loop section into the matching rules file, tailored to your repo:

We've found that adding a short test-coverage loop to your agent's instructions is the most fruitful way to use verifAIed. I want you to write that loop into this repo's agent-instructions file, tailored to how this repo actually runs its tests.

verifAIed is a coverage analysis tool. Its CLI (`verifaied upload`) pushes a local `coverage.json` + `junit.xml` to the server; its MCP server exposes a `check_done` tool that returns a `{done, status, gaps}` verdict for the branch (the loop's stop condition), and a `fix_branch` tool that returns a single prompt describing every untested function, partial branch, and failing test. The agent loop is: run tests → `verifaied upload` → call `check_done` → if `done` is true stop, otherwise fix the gaps (`get_baseline_prompt` per function, or `fix_branch` for one prompt covering everything) → repeat.

Do the following, in order:

# 1. Detect the project setup

- **Test runner & command**: look at `pyproject.toml` (`[tool.pytest.ini_options]`, `[project.scripts]`), `pytest.ini`, `tox.ini`, `setup.cfg`, `Makefile` targets (`test`, `check`), `justfile`, or a top-level `scripts/` directory. Use the test command the project already uses — don't invent a new one.
- **Coverage flags**: pytest must produce `coverage.json` with branch coverage AND per-test contexts, plus `junit.xml`. If the existing command already does that, reuse it. Otherwise build the command:

pytest --cov --cov-branch --cov-context=test --junitxml=junit.xml coverage json --show-contexts -o coverage.json

The `coverage json --show-contexts` step is non-negotiable — pytest-cov's `--cov-report=json` alone drops the per-test contexts, which makes verifAIed mark the upload "Needs attention".
- **Agent-instructions file**: pick the one that matches the agent you (the model) are. If unsure, fall back to the project's existing convention.
- Claude Code: `CLAUDE.md`
- Codex CLI / OpenAI: `AGENTS.md`
- Cursor: `.cursorrules` or `.cursor/rules/*.mdc`
- Gemini CLI: `GEMINI.md`
- GitHub Copilot: `.github/copilot-instructions.md`
- If none exists in the repo, create the one matching your own agent.

# 2. Verify the CLI is installed and configured

- Run `verifaied --help`. If it exits non-zero, install it: `uv tool install verifaied` (preferred) or `pipx install verifaied`.
- Check `echo $VERIFAIED_API_TOKEN`. If empty, stop and tell the user to mint one from the verifAIed app (Settings → Tokens) and export it: `export VERIFAIED_API_TOKEN=vr_live_...`. Do NOT invent a placeholder and do NOT proceed without it.

# 3. Write the loop into the agent-instructions file

Append (or create) a section titled `## Test coverage loop with verifAIed` containing:

- The exact test command(s) you built in step 1 — so future runs of the agent don't have to re-derive them.
- The exact upload command (`verifaied upload`, plus any flags this repo needs — e.g. `--coverage path/to/coverage.json` if your test command lands the JSON outside the repo root).
- An instruction to call the `check_done` tool on the verifAIed MCP server after each upload, and to stop only when it returns `done: true`.
- An instruction that while `check_done` reports gaps, fix them: call `get_baseline_prompt(function_id)` for a free per-function test prompt, or `fix_branch` with `scope: everything` for a single prompt covering every gap, then apply it.
- An instruction to repeat from step 1 until `check_done` reports `done: true`.
- One line: "If `VERIFAIED_API_TOKEN` is unset, stop and ask the user — do not skip this loop."

# 4. Confirm with the user

Show the user the diff of the file you edited (or created) and the test command you wrote in. Do NOT run the loop yourself yet — let the user review first.

# Hard rules

- Do NOT change the test command's behavior beyond adding coverage / junit flags.
- Do NOT add new dependencies if pytest-cov and coverage are already transitive deps.
- Do NOT write a fallback that skips the loop when the token is missing — the agent must stop and ask.
- Do NOT enumerate functions yourself or write tests; `check_done` lists the gaps and `fix_branch` / `get_baseline_prompt` return the prompts that close them.
- Do NOT decide "done" yourself; `check_done` is the stop condition.
- Do NOT replace an existing agent-instructions file — append a new section.

Or paste the instructions in by hand

If you'd rather skip the meta-prompt, drop this section into your agent's rules file as-is. Swap in your repo's actual test command if it differs from the pytest default:

## Test coverage loop with verifAIed

After implementing or modifying any code in this repo, run this loop until verifAIed reports done:

1. Run the test suite with branch coverage and per-test contexts:

pytest --cov --cov-branch --cov-context=test --junitxml=junit.xml coverage json --show-contexts -o coverage.json

2. Upload the results to verifAIed:

verifaied upload

3. Ask if you're done: call the `check_done` tool on the verifAIed MCP server (or run `verifaied check-done`). It returns `{done, status, gaps}` for this branch — failing tests plus untested/partially-covered functions among the ones you changed.
4. If `done` is true, stop — the branch is covered.
5. Otherwise fix the gaps it lists: call `get_baseline_prompt(function_id)` for a free test prompt per function, or `fix_branch` with `scope: everything` for a single LLM-written prompt covering every gap on the branch.
6. Go back to step 1.

Stop only when `check_done` reports `done: true`. If `VERIFAIED_API_TOKEN` is not set, stop and ask the user — do not skip this loop.