agentic-test-forge

Python quality enforcement for AI-generated and legacy codebases. Implements Uncle Bob Martin's workflow: CRAP analysis, mutation testing, and Gherkin scenario mutation, optimized for agentic development and CI gates.

Status

Latest PyPI: 1.1.0 — mutmut 3.5 runner, CLI config paths, CRAP relative coverage keys (#135). v1.1 product work (#58) shipped earlier; this tag is the pin consumers should use.

Install

pip install agentic-test-forge

Pin a version:

pip install agentic-test-forge==1.1.0

For local development of this repo:

pip install -e ".[dev]"

Alternative (VCS install):

pip install "agentic-test-forge @ git+https://github.com/cheezd/agentic-test-forge.git"

Usage

forge --help
forge crap --threshold 30
forge mutate --base main --threshold 80
forge mutate-gherkin --base main --threshold 80
forge check

Omitted --path / --features-path uses [tool.forge].paths and gherkin_paths. Repeat --path to override with multiple roots (forge check --path dashboards --path ghdash).

Run tests with coverage, then the full quality gate:

pytest --cov=src
forge check --json report.json

Differential mutation uses git diff against --base (default main) and skips unchanged files tracked in .forge/mutation-manifest.json. Use --full to ignore the manifest.

Gherkin mutation mutates Examples table cells in changed .feature scenarios, runs the configured acceptance test command, and tracks results in .forge/gherkin-manifest.json.

Thresholds are gate cutoffs, not comparable scales: crap_threshold is a maximum CRAP score per function; mutation_threshold and gherkin_threshold are minimum mutation kill rates (0–100%). See score interpretation for what the numbers mean.

Configure per-project thresholds in pyproject.toml:

[tool.forge]
paths = ["src"]
crap_threshold = 30
crap_formula = "standard"  # standard | simplified
manifest_dir = ".forge"
mutation_threshold = 80
mutation_base_ref = "main"
mutation_test_cmd = "pytest"
gherkin_threshold = 80
gherkin_base_ref = "main"
gherkin_test_cmd = "python -m behave"  # not bare `behave` — often missing from PATH
gherkin_runner = "behave"  # behave | pytest
gherkin_paths = ["features"]

[tool.forge.gates]
crap = true
mutation = false
gherkin = false
dry = true         # advisory — does not fail forge check

Optional local override: you do not need forge.toml for normal use — [tool.forge] in pyproject.toml is enough (including consumer repos). If present, a forge.toml in the current working directory is merged on top of pyproject.toml (useful for uncommitted experiments, e.g. stricter thresholds on your machine). Unlike pyproject.toml, forge does not search parent directories for forge.toml; run from the directory that contains it, or rely on pyproject.toml only.

Staged rollout for legacy repos: enable crap first, then advisory dry, then mutation on Linux CI or WSL (keep it off in Windows pre-commit), then gherkin once python -m behave (or pytest-bdd) is already green. See consumer-ci staged rollout.

Consumer CI integration: see docs/consumer-ci.md (GitHub Actions, version pinning, Django / monorepo, Gherkin, Windows / WSL mutation, Windows console notes).

Pre-commit (optional)

Run forge check locally before commit (respects [tool.forge.gates]):

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/cheezd/agentic-test-forge
    rev: v1.1.0
    hooks:
      - id: forge-check

pip install pre-commit agentic-test-forge==1.1.0
pre-commit install
pytest --cov=src   # CRAP gate needs .coverage
pre-commit run forge-check --all-files

See consumer-ci — pre-commit for coverage prerequisites, Windows/mutation notes, and troubleshooting.

Development

pip install -e ".[dev]"
pytest
ruff check .
mypy src

Domain language

See docs/domain/CONTEXT.md.

Architecture decisions

Index and when to write ADRs: docs/adr/README.md. Package layout, dependency direction, and refactor conventions: docs/adr/0001-package-boundaries-and-refactor-conventions.md.

License

Licensed under the GNU Lesser General Public License v3.0 or later (LGPL-3.0-or-later).

What this means in practice:

• Modifications to forge must be shared under LGPL when you distribute them.
• Using forge to check your code — via CLI in CI, locally, or on build servers — does not require your application or SaaS product to become open source.
• Importing forge as a library in a proprietary product is generally permitted under LGPL (unlike GPL), subject to LGPL’s requirements (e.g. allowing replacement of the library).

We deliberately use LGPL, not AGPL, so network/SaaS deployment of your product does not trigger additional copyleft beyond the library itself. This is not legal advice; consult counsel for your specific deployment.

Roadmap

v1.0 (shipped)

1. Foundation & CLI shell — done
2. CRAP analyzer (radon + coverage.py) — done
3. Differential code mutation (mutmut) — done
4. Gherkin mutation — done
5. Quality gate orchestrator (forge check) — done
6. DRY flagging (advisory) — done

v1.1 (shipped — #58)

• PyPI publish & GitHub Release — done (#64)
• Dogfood CI (forge check + report artifact) — done (#70)
• External consumer pilot (compliance-llm) — done (#71)
• Pre-commit hook — done (#74)
• Docs polish & ADR bootstrap — done (#78–#81, #128)

Beyond v1.1

• Semantic DRY (#122) — deferred to v1.2+