Sentry

Português | English

Change-oriented test quality CLI. sentry creates the spec folder, validates the case matrix in markdown, runs the suite, reads the diff and the coverage, and issues an auditable verdict — and writes the workflow down for the AI agent that will implement the change.

Markdown is the source of intent: the CLI parses and validates CASES.md and PROMPT.md — it never writes them. The AI agent does the drafting; Sentry guarantees structure, traceability and a verdict.

Not to be confused with getsentry's Sentry (error monitoring). This project is distributed as sentry-test.

The division of responsibility

• The AI agent declares intent — writes CASES.md: requirement, layer, type, priority, input, expected result.
• Sentry measures reality — decides status, associated test, evidence and verdict.

The agent never writes status. Sentry never calls a model to reach a verdict. That's what makes the verdict auditable and reproducible. The optional video commands (promo and training) are the exception: they hand the work to the agent, which spends tokens, and they never touch the verdict.

Installation

Using a virtual environment (venv) is recommended: it keeps Sentry's dependencies apart from the rest of your machine and from other projects.

python -m venv .venv
source .venv/bin/activate      # Windows: .venv\Scripts\activate
pip install sentry-test

Requires Python 3.11+. Check with sentry --version (prints the package version).

If pytest or coverage are missing, init warns you. To install them too:

sentry init --install

Setup

cd your-project
sentry init

Creates .sentry/ (specs, runs, reports and database), sentry.toml, the .gitignore entries, and writes the workflow in two places:

• sentry-cases skill (SKILL.md), written to the folder(s) declared in [init] skills_dirs in sentry.toml — .claude/skills/ by default; --skills-dir picks another. An agent that reads skills from that folder loads it automatically.
• AGENT-SENTRY.md at the project root — the same workflow in plain markdown, without depending on any tool's convention.

For other agents (Cursor, Windsurf, Codex, opencode), point them to AGENT-SENTRY.md from the rules file each one already uses — for example, a line in .cursor/rules or AGENTS.md:

To write or review test cases, follow AGENT-SENTRY.md.

init is idempotent: it doesn't erase history, doesn't overwrite existing configuration, doesn't duplicate structures.

The workflow

1. sentry new "customer registration" — creates .sentry/specs/customer-registration/ with PROMPT.md (request preserved) and a blank CASES.md
2. sentry-cases — your agent asks about anything ambiguous and fills in CASES.md following the template
3. sentry check customer-registration — is the structure valid? are the catalog's equivalence classes covered?
4. your agent links each case to the real test with the # scenario: <exact case name> marker
5. sentry watch — leave it running while you write: it re-evaluates on every save without executing tests, and escalates to the full run when the file you saved is a test
6. sentry context --json — when a gap shows up, your agent works over the gaps instead of re-reading the code
7. sentry review — check + run + report in one command, as a local pre-commit hook
8. sentry status — periodically, the whole application instead of just the diff: which files no test reaches at all
9. sentry report / sentry history — read back and compare between runs

Every step also works without an agent, via the commands below.

Commands

Exit code 0 on success; see the exit code table further down.

Two modes, by design

The mode decides what the report is allowed to claim about coverage.

• Instant — no test process is started. It reports structure, traceability, orphan markers, the diff, and the coverage of the last full run. Reused coverage is always stamped as coming from that run, with its timestamp, never as measured now. With no previous full run, coverage comes out unavailable and no number is asserted.
• Full — executes the suite. It's the mode for the pre-commit hook and for anything that needs a count of its own.

sentry watch picks between them on every save: instant for source files, full when the saved file is a test.

A full run reuses the previous one when nothing that matters has changed. The cache key is a content hash of three inputs and nothing else: the files the diff lists, the test files, and the specs. Change any of them and the suite runs again — either the code is different, or what's being asked of it is. A run that comes back from the cache says so in the report, with the id and the timestamp of the run the evidence came from: reusing without saying it would turn old evidence into a new claim.

The gap payload

sentry context --json is the loop's other half: the agent stops reading the code and reads what has no evidence. Keys: summary, uncovered_lines, uncovered_error_paths, failing_tests, scenarios_without_tests, missing_equivalence_classes, orphan_markers, limitations.

Every key always exists, empty when there's nothing — an absent key would be ambiguous between "nothing to do" and "Sentry didn't look". Nothing there is recalculated: each gap was already decided by whatever held the data, and the payload only crops it, so it can never disagree with the report.

Cost

Every report ends with a ## Custo block: the tokens Sentry spent — zero, because no model is called to reach the verdict (the optional video commands hand their work to the agent and stay outside this block), which is what makes the same commit always produce the same verdict — and the size of what it summarized, in bytes: the diff it read and the report itself. The conversion to tokens is declared as an estimate at 4 characters per token, with no real tokenizer; it's an order of magnitude for sizing a context budget, not an exact count.

Exit codes

Four distinguishable states, to separate "poorly tested code" from "my environment broke":

An infrastructure error never produces an approved verdict: a suite that failed to run is different from a suite that failed.

sentry check keeps its own semantics: 0 valid structure, 1 structural errors, 2 couldn't resolve the spec.

Skills

Generated for each configured agent; AGENT-SENTRY.md covers the rest.

Deterministic rules

Ten rules, with severity configurable per project.

Without a declared threshold, Sentry doesn't invent a minimum. The project decides what's "enough", and the report records the number applied.

Equivalence class catalog

A fixed table of situations that need a test, per field type. It doesn't generate cases: it flags the ones the agent left undeclared.

Known types: cpf, cnpj, email, senha (password), data (date), telefone (phone), cep (postal code), inteiro (integer), decimal, texto (text), rota (route), formulario (form), navegacao (navigation), responsivo (responsive), acessibilidade (accessibility) — the last four are for frontend-layer cases verified by Playwright evidence, not line coverage.

A class that doesn't make sense for a field can be dismissed with a justification, instead of becoming an artificial case or an eternal complaint:

## Classes não aplicáveis

- **exclude/tamanho-maximo-excedido**: it's a configuration parameter, not a form field

The dismissal removes the finding, but it stays recorded in the report — nothing disappears silently.

Coverage dimensions

Each one reports covered, partial, not covered or not applicable, with evidence.

not applicable is distinct from not covered: a project with no routes isn't penalized on the security dimension.

Configuration

sentry.toml at the root, versionable, with no secrets. Everything beyond what init already writes is optional.

[project]
name = "my-project"

[specs]
path = ".sentry/specs"

[test]                    # any runner that exports JUnit XML
command = "npx jest"
junit_xml = "reports/junit.xml"   # required outside pytest: it's the only source of counts

[tests]                   # where to look for tests
paths = ["tests"]         # default: tests, test, spec, __tests__

[coverage]                # report generated by your own suite
path = "coverage/lcov.info"
format = "lcov"           # optional: detected by content when omitted

[e2e]                     # optional: second suite for frontend-layer cases
command = "npx playwright test --reporter=junit"
junit_xml = "frontend/reports/junit.xml"
paths = ["frontend/e2e"]

[video]                   # optional: the agent that runs the brag (default: Claude)
agente = ["meu-agente", "--rodar", "{prompt}"]   # `{prompt}` marks where the request goes

[init]                    # optional: where the sentry-cases skill is written (default: .claude/skills)
skills_dirs = [".claude/skills", ".agentes/skills"]   # one or more folders; [] writes no skill

[analysis]
run_tests_by_default = false
timeout_seconds = 300
exclude = ["frontend/"]   # directories out of the analysis scope

[policy.thresholds]       # without this, no minimum is enforced
changed_coverage = 85
global_coverage = 90

[policy.severities]       # overrides the severity of any rule
coverage-missing = "high"

[catalog.fields]          # field types from your own domain
matricula = ["empty", "invalid-format", "valid"]

[dimensions]              # axes that don't apply to the project
disabled = []

.sentry/ holds specs, runs, reports and the database. It stays out of Git, with one deliberate exception: the current report, .sentry/reports/latest-<date>.md, is versioned, so the verdict shows up in the PR diff without the reviewer having to run Sentry. The name carries the date and time of the run, and there is only ever one: the previous one is replaced when a new run writes.

History is kept indefinitely, and grows with every run. To prune it:

sentry clear --keep-last 10        # shows what would be removed
sentry clear --keep-last 10 --yes  # removes it

Specs are never removed: they're declared intent, not generated evidence.

Supported stacks

Derivation — from the request to the case matrix — is language-agnostic: CASES.md is markdown and the catalog reasons about data type, not code.

Verification depends on the exchange format your suite exports, not on the tool:

Error path detection outside Python is less precise than AST, and the report records that difference as a limitation — it never hides it.

In Python, pytest is the only runner instrumented automatically: Sentry recognizes it as pytest, python -m pytest, and the venv's executable, wraps it in coverage run and collects the count on its own. In Django, prefer command = "python -m pytest" with pytest-django over manage.py test. Any other command — including python -m unittest — runs exactly as declared, with no flag injected: to measure it, declare the junit_xml your suite generates, otherwise the verdict comes out not executed for lack of evidence.

The frontend layer is verified by a second suite, declared under [e2e] — Playwright's JUnit reporter is a native fit. Its cases never get line coverage: status comes from execution evidence (the scenario ran and passed, trace/screenshot attached), never from a coverage number, which the report states as a declared limitation.

Local-first

No telemetry, no external calls, no code or diff ever sent anywhere. All history stays on the machine.

Development

python -m pip install -e .
python -m pytest

Sentry analyzes itself: sentry run --spec all --run-tests at the root of the repository matches the cases declared in .sentry/specs/ against the real test functions and reports the five dimensions.

License

MIT