spec-trace

A traceability lint between behavioural specs and tests. Zero dependencies, one command, works as a CI gate.

Spec points are markdown bullets that end in a keycode. Tests cite the keycodes they cover. spec-trace reads both sides and reports the drift.

- Results can be filtered to a date range. Both boundaries are inclusive. `[date-filter-range]`

# SPEC: warehouse/querying#date-filter-range
def test_filters_by_date_range() -> None: ...

$ spec-trace
OK

Install

uv tool install spec-trace     # or: uvx spec-trace

Python 3.11+. No runtime dependencies — the stdlib-only single package is the distribution story.

Getting started

spec-trace init     # scaffold specs plus the agent instructions
spec-trace          # lint
spec-trace docs     # the full convention

init writes a specs directory with a landing page, a Claude Code skill, and an AGENTS.md section. The convention is more than half the product: the checker is useless without the writing discipline, and its main consumer is coding agents.

Two flags change what it writes. --example adds a starter spec file to copy the format from; it is opt-in because no test cites its keycodes, so spec-trace --warn reports them as coverage gaps until you replace it. --skill-only writes the skill alone, for a repo that already has its own specs. The two are mutually exclusive, as is --global.

What the gate fails on

spec-trace hard-fails on referential integrity only:

Failure	Meaning
Orphaned ref	a test cites a spec ID that does not exist
Malformed ref	a SPEC: token with no #keycode
Duplicate ID	the same keycode twice in one spec file
Over-long bullet	behaviour text above the limit (default 100 chars)

Coverage gaps — untested spec points and unspecified tests — are advisory, shown under --warn.

This split is the reason the tool survives as a gate. The failures above are cheap to fix and objectively wrong. A gate that failed on coverage would be deleted within a month. In the codebase this tool was extracted from, the gate has stayed green since it was enforced in CI, across 167 spec files and 3,602 spec points.

Orphaned refs get a fuzzy-match suggestion, because a mistyped keycode is the most common failure:

ERROR: 1 orphaned test references
  A test cites a spec ID that does not exist. Fix the ID, or add the spec point.
  tests/querying_test.py:
    - SPEC: warehouse/querying#date-filter-rnage (line 42)
        did you mean `warehouse/querying#date-filter-range`?

Usage

spec-trace                    # lint the repo; exit 1 on a gate failure
spec-trace warehouse/         # limit the report to one subsystem
spec-trace --warn             # include advisory coverage sections
spec-trace --warn-unspecified # as --warn, plus a list of each unspecified test
spec-trace --missing          # list untested spec points
spec-trace --ok               # list tested spec points
spec-trace --orphan           # list orphaned references
spec-trace --json             # machine-readable output
spec-trace --root path/to/repo   # scan this repo root instead of auto-detecting
spec-trace --version          # print the version
spec-trace init --update      # refresh generated instruction files
spec-trace init --example     # also write a starter spec file
spec-trace init --skill-only  # write only the Claude Code skill
spec-trace init --global      # install the skill into ~/.claude instead

--warn-unspecified implies --warn: it prints the advisory sections and names each unspecified test, where --warn alone reports only how many files contain them. Reach for it when the summary count is not enough and you need to act on the individual tests.

--root skips the auto-detection that walks upward for a config file or a .git directory. It is what you want when the tool runs from outside the repo it should scan. init accepts it too.

A path filter narrows what is reported, not what is parsed, so a filtered run detects the same orphaned references as a full run. Failures are scoped by the spec path they belong to, so a filtered run reports only those inside the filter — except a malformed reference, which names no spec path and is therefore always reported.

In CI

- name: Spec check
  run: uvx spec-trace

Configuration

Config is optional. Zero config works for a Python or TypeScript repo keeping specs in docs/specs. Settings live in [tool.spec-trace] in pyproject.toml, or in spec-trace.toml for non-Python repos.

[tool.spec-trace]
specs_dir = "docs/specs"
max_behaviour_length = 100
exclude_dirs = ["node_modules", ".venv", "dist", "build", "__pycache__", ".git"]
exclude_files = ["tests/fixtures_test.py"]   # files whose SPEC: text is data, not a reference

[[tool.spec-trace.languages]]
name = "php"
test_glob = "*Test.php"
comment = "//"
test_function = 'function\s+(test\w+)'

[tool.spec-trace.test_types]   # optional; path prefix -> label in coverage output
"integration-tests/" = "e2e"

Every list setting replaces its default rather than adding to it. Setting exclude_dirs drops the built-in exclusions, so repeat the ones you still want. Defining any language likewise replaces both built-in languages.

exclude_files skips a matching test file entirely, so a real SPEC: reference inside it is ignored too. Reach for it only when a file contains reference-shaped text as data — a better fix is usually to build that text from parts, as tests/spec_fixtures.py does here.

For coding agents

An agent meets this tool at three moments, and each has its own surface:

1. While planning — init installs repo instructions (a Claude Code skill plus an AGENTS.md section) so specs get written with the work, not patched in after a gate failure.
2. On first contact — --help carries a compressed convention: both formats, one example of each, and what the gate fails on.
3. When the gate fails — every ERROR states the fix, and the FAIL line points at spec-trace docs.

The projected files are deliberately thin: the stable format cheat-sheet inline, everything else deferred to spec-trace docs, whose content ships inside the package and versions with it. init --update rewrites the skill, and rewrites the AGENTS.md region between marker comments, leaving your own text around it alone. It never touches the specs README or a starter spec written by --example: init asks you to describe your app in one and to replace the other, so both are yours after the first write.

Known limits

Being honest about what this does not do:

• It checks linkage, not truth. Nothing verifies that a test actually exercises the behaviour its bullet describes. That is delegated to review and agent discipline.
• The test→spec direction is weak at scale. A large codebase adopting this will have thousands of unspecified tests, a number nobody acts on. Treat it as a backlog signal, not a target.
• Parsing is regex, not AST. The "SPEC comment attaches to the next test function" rule is a loose heuristic, and TypeScript describe nesting is discarded. Acceptable for a lint.
• A green gate is not coverage. A spec point counts as tested when any test cites its keycode. Nothing checks that the test exercises the behaviour, or that it asserts anything at all. A repo with every spec point cited and every assertion deleted still passes. Read a green run as "the links resolve", not "the behaviour works" — including for this repo's own gate.

Non-goals: no semantic verification, no test running, no coverage measurement, no MCP server, and no runtime dependencies.

Development

cp .env.example .env           # only needed for a release, but see below
uv sync --extra dev --locked   # run `uv lock` if this reports a stale uv.lock
bin/check                      # every gate CI runs; reports all failures, not just the first
bin/check --only lint          # one gate by name; --list names them all

Copy .env.example even when you are not releasing: a shell that sets UV_ENV_FILE=.env fails every uv run with No environment file found until the file exists. An empty token is fine for the gates.

bin/check is the single definition of the gate commands: it is what a contributor runs before committing, what bin/publish runs before releasing, and what CI runs one gate per job. That is why no gate command is written out below — the table names each gate, not its command.

Gate	Checks	Local	CI job
format	Formatting matches ruff format. Run uv run ruff format . to rewrite.	bin/check --only format	gate (format)
lint	Ruff lint rules, including unused imports.	bin/check --only lint	gate (lint)
typecheck	Pyright over src/ and tests/, basic mode.	bin/check --only typecheck	gate (typecheck)
deadcode	Vulture finds module-level symbols nothing references.	bin/check --only deadcode	gate (deadcode)
test	The pytest suite.	bin/check --only test	test (3.11), test (3.13)
spec-check	The tool gates its own specs, with --warn for coverage drift.	bin/check --only spec-check	gate (spec-check)

Tests run on 3.11 and 3.13 in CI; bin/check runs them once, on your interpreter. A contributor on 3.12 who is green locally can still be red on 3.11, so a passing bin/check is necessary but not sufficient. The other five gates are single-interpreter and match CI exactly.

CONTRIBUTING.md covers the rest, including the house rule a drive-by contributor cannot guess: this repo gates its own specs, so a pull request adding a test with no SPEC: reference fails CI. Changes worth a release note go in the Unreleased section of CHANGELOG.md.

spec-trace uses its own convention: see docs/specs/, with the cross-references in tests/test_spec_linked.py, and behaviour below the level of a spec point covered in tests/test_internals.py.

Release

bin/publish [--minor|--major] [--dry-run]

That is the whole release. It rebases onto origin/main, runs bin/check, writes the new version, retitles the changelog, commits, builds, publishes to PyPI, tags, and pushes the commit and the tag. Needs UV_PUBLISH_TOKEN in the environment or in .env, a clean working tree, a non-empty Unreleased section in CHANGELOG.md, and mael on PATH for the rebase.

--dry-run runs the gates and builds without publishing, tagging or rebasing, then reverts the version bump. A failed upload rolls the release commit back, so a re-run starts from the same version number rather than skipping one.

Related

maelstrom is a sibling tool for orchestrating parallel agent development. The two share a distribution pattern but have no dependency in either direction: spec-trace is repo-scoped and must run in CI with nothing but Python.

Licence

MIT