mentistest-coverage 0.4.0

Analyze test strategy across SaaS products with coverage metrics

mentistest-coverage

Strategic test-suite analyser. Looks at a Python project's source tree, test tree, JUnit XML and (optionally) an OpenAPI spec, and produces a test-strategy report — not just a coverage percentage.

What it does that coverage.py doesn't:

• Classifies every test as unit / integration / contract / e2e using markers, imports, fixtures and directory layout.
• Weights coverage gaps by cyclomatic complexity and call-graph depth so the 20 functions that matter rank above the 200 that don't.
• Correlates tests against the OpenAPI spec to flag uncovered endpoints.
• Tracks results over time in a local SQLite history store with full git/CI provenance, and renders a trend dashboard per project.
• Detects regressions between snapshots and can fail CI when a previously- covered symbol loses coverage.
• Ships an opt-in AI second opinion (Claude Haiku → Sonnet escalation, gated to skip healthy projects — typical cost <£0.01/run).
• Exports HTML (interactive), JSON (machine-readable) and PDF (board-ready) reports.
• Ships a first-class GitHub Action that posts a coverage-delta PR comment and uploads artifacts.

Local-first. The analyser runs against filesystem paths; source code is never uploaded anywhere. The optional AI call sends a structured summary (file paths, coverage numbers, test names) — never the source itself.

Install

pip install mentistest-coverage

…or from a clone:

pip install -e .

Verify:

mentistest --help

Every PyPI release is built via OIDC trusted publishing and signed with Sigstore — see RELEASING.md for the full chain and verification recipe.

Quick start

A complete analysis with all the features wired in:

mentistest analyze my-service \
    --source ./src --tests ./tests \
    --junit ./reports/junit.xml \
    --api-spec ./openapi.yaml \
    --output-json ./report.json \
    --output-html ./report.html \
    --output-pdf  ./report.pdf \
    --track --compare-baseline --fail-on-regression \
    --detect-git \
    --ai

What that does:

Flag	Effect
--config / -c	Project manifest (mentistest.yaml); auto-detected when omitted
--source / -s	Source directory (repeatable)
--tests / -t	Test directory (repeatable)
--junit	JUnit XML report path (repeatable)
--api-spec	OpenAPI / Swagger spec — drives endpoint-coverage analysis
--coverage-xml	Real coverage.xml from coverage.py (preferred over the fallback)
--coverage	Real coverage report in any format (Cobertura, JaCoCo, Istanbul, SimpleCov, LCOV, Go) — auto-detected
--coverage-format	Format of --coverage (auto default, or an explicit format)
--per-test-coverage	Per-test coverage (Jest/Vitest Istanbul JSON map or a dir of Go labelled coverprofiles) — populates "tests touching this file" for JS/TS and Go
--per-test-format	Format of --per-test-coverage (auto default, jest, vitest, go)
--output-json / -j	JSON report path
--output-html / -H	Interactive HTML dashboard path
--output-pdf / -P	Board-ready PDF report path
--track / --no-track	Save the run as a snapshot in the history DB
--db-path	Override ~/.test-coverage/history.db
--compare-baseline	Diff this run against the most recent snapshot
--regression-threshold N	Allowable coverage drop in pp (default 0)
--fail-on-regression	exit 1 if a regression is detected
--detect-git / --no-detect-git	Capture commit, branch, CI run ID, provider
--ai	Run the optional AI commentary (requires ANTHROPIC_API_KEY)
--test-type	Default type label when no signal fires (unit/integration/contract/e2e)
--verbose / -v	Debug logging

Multi-repo projects: the manifest

When different types of test live in different places — unit tests in-repo, an e2e suite in a sibling repo, a Pact contract suite elsewhere — declare them once in a checked-in mentistest.yaml and run mentistest analyze with no other flags:

project: my-service
source: [src]
tests: [tests/unit, tests/integration]
external:
  - path: ../e2e-suite
    type: e2e
    attribution: endpoint        # match URL literals against the OpenAPI spec
    junit: [../e2e-suite/reports/junit.xml]
  - path: ../contract-tests
    type: contract
    attribution: none

CLI flags take precedence over manifest scalars; test locations are merged. Paths resolve relative to the manifest. See the manifest guide for the full schema.

Server mode

mentistest serve --host 0.0.0.0 --port 8000

Because the /analyze endpoint reads arbitrary filesystem paths, serve refuses to bind a non-loopback host (anything other than 127.0.0.1/ ::1/localhost) unless TEST_COVERAGE_API_KEY or TEST_COVERAGE_ALLOWED_PATHS is configured. Pass --insecure to override on a trusted private network. The published Docker image binds 0.0.0.0 and sets TEST_COVERAGE_REQUIRE_SECURITY=1, so the container likewise won't start unprotected.

Then:

• http://localhost:8000/docs — interactive OpenAPI docs for the REST API
• http://localhost:8000/benchmarks — multi-project benchmark dashboard
• http://localhost:8000/trends/<project> — single-project trend dashboard
• http://localhost:8000/report/html?project_name=<name> — last-rendered report
• http://localhost:8000/report/pdf?project_name=<name> — last-rendered PDF
• http://localhost:8000/history/<project> — JSON list of snapshots
• http://localhost:8000/regression/<project> — JSON regression diff

GitHub Action

- uses: HarryDouglas/test_coverage_tool@v1
  with:
    source_dirs: src
    test_dirs: tests
    junit_xml_paths: reports/junit.xml
    track_history: 'true'
    fail_on_regression: 'true'

Posts a PR comment with the coverage delta, uploads JSON + HTML reports as workflow artifacts, and fails the check if a symbol loses coverage. See action.yml for all inputs.

Action vs PyPI wheel: the action is its own distribution channel — referenced by git tag, not installed from PyPI. Under the hood it installs the wheel from PyPI and runs the CLI, so the two move together but version independently. The action's @v1 tag pins a major; minor / patch wheel updates roll through without a workflow change.

Security

If you're deploying the HTTP API beyond localhost, read SECURITY.md. Two env vars (TEST_COVERAGE_API_KEY and TEST_COVERAGE_ALLOWED_PATHS) enable bearer-token auth and filesystem path-traversal protection. Both are off by default so local, loopback-only use is unaffected — but the tool fails closed the moment it would be network-reachable: serve rejects a non-loopback --host (unless --insecure), and create_app refuses to start when TEST_COVERAGE_REQUIRE_SECURITY=1 (set in the Docker image) is configured without a protection layer.

To report a vulnerability, email [e-mail] — don't open a public issue.

Public benchmark suite

Benchmarking is built around a frozen corpus — a small set of checked-in fixtures (benchmarks/corpus/, plus the polyglot monorepo fixture) that ship their own source and pre-captured coverage / JUnit / OpenAPI artefacts. The analyser runs over them offline and deterministically, so the default make benchmark is fast and never clones, installs, or runs a foreign test suite. The corpus is the single foundation for three things:

• Regression gate — a golden report.json per member, diffed in CI on every change (unittests/benchmarks/test_corpus_golden.py).
• Accuracy check — hand-verified ground truth per member (unittests/benchmarks/test_corpus_truth.py).
• Showcase dashboard — the same reports, rendered fast.

make benchmark         # render the frozen corpus (fast, offline, deterministic)
make benchmark-serve   # serve the dashboard at localhost:8003/benchmarks
make benchmark-check   # fail if any committed report is from an older tool version

make benchmark-refresh # the slow, occasional run against real OSS repos
                       # (clone + install + run their suites) — a real-world
                       # smoke test, not the default

Outputs land in reports/benchmarks/<project>/ (HTML + JSON). After an intentional metric change, regenerate the goldens with UPDATE_CORPUS_GOLDENS=1 pytest unittests/benchmarks/test_corpus_golden.py and commit them. See benchmarks/corpus/README.md for the corpus model and how to add a member.

Library use

from pathlib import Path
from test_coverage_tool.core.analyzer import AnalysisEngine
from test_coverage_tool.core.git import detect_git_metadata
from test_coverage_tool.reports.html_generator import HTMLReportGenerator
from test_coverage_tool.reports.json_generator import JSONReportGenerator
from test_coverage_tool.history.store import HistoryStore

engine   = AnalysisEngine()
analysis = engine.analyze(
    project_name     = "my-service",
    source_dirs      = [Path("src")],
    test_dirs        = [Path("tests")],
    junit_xml_paths  = [Path("reports/junit.xml")],
    api_spec_path    = Path("openapi.yaml"),
    git_metadata     = detect_git_metadata(),
)
JSONReportGenerator().write(analysis, Path("report.json"))
HTMLReportGenerator().write(analysis, Path("report.html"))

# Persist a snapshot
HistoryStore(Path("history.db")).save("my-service", analysis_to_dict(analysis))

Configuration

Environment variables:

Variable	Purpose
ANTHROPIC_API_KEY	Enables --ai (otherwise the flag is a no-op)
TEST_COVERAGE_DB_PATH	Override the default ~/.test-coverage/history.db location
MENTIS_BENCH_AI	Set to 1 to opt-in to AI commentary on make benchmark

Development

make install     # install dev deps via uv
make test        # pytest --cov on unittests/
make lint        # ruff + mypy --strict + pylint
make benchmark   # rebuild the public benchmark dashboard

The repo uses ruff, mypy (--strict), pylint and pytest. The quality bar: zero ruff/mypy/pylint issues, pytest -x -q must pass before merge.

License

MIT — see LICENSE.