spec-trace

A traceability lint between behavioural specs and tests. Zero dependencies, one command, works as a CI gate.

Spec points are markdown bullets that end in a keycode. Tests cite the keycodes they cover. spec-trace reads both sides and reports the drift.

- Results can be filtered to a date range. Both boundaries are inclusive. `[date-filter-range]`

# SPEC: warehouse/querying#date-filter-range
def test_filters_by_date_range() -> None: ...

$ spec-trace
OK

Install

uv tool install spec-trace     # or: uvx spec-trace

Python 3.11+. No runtime dependencies — the stdlib-only single package is the distribution story.

Getting started

spec-trace init     # scaffold specs plus the agent instructions
spec-trace          # lint
spec-trace docs     # the full convention

init writes a specs directory with a landing page, a starter spec file, a Claude Code skill, and an AGENTS.md section. The convention is more than half the product: the checker is useless without the writing discipline, and its main consumer is coding agents.

What the gate fails on

spec-trace hard-fails on referential integrity only:

Coverage gaps — untested spec points and unspecified tests — are advisory, shown under --warn.

This split is the reason the tool survives as a gate. The failures above are cheap to fix and objectively wrong. A gate that failed on coverage would be deleted within a month. In the codebase this tool was extracted from, the gate has stayed green since it was enforced in CI, across 167 spec files and 3,602 spec points.

Orphaned refs get a fuzzy-match suggestion, because a mistyped keycode is the most common failure:

ERROR: 1 orphaned test references
  A test cites a spec ID that does not exist. Fix the ID, or add the spec point.
  tests/querying_test.py:
    - SPEC: warehouse/querying#date-filter-rnage (line 42)
        did you mean `warehouse/querying#date-filter-range`?

Usage

spec-trace                    # lint the repo; exit 1 on a gate failure
spec-trace warehouse/         # limit the report to one subsystem
spec-trace --warn             # include advisory coverage sections
spec-trace --missing          # list untested spec points
spec-trace --ok               # list tested spec points
spec-trace --orphan           # list orphaned references
spec-trace --json             # machine-readable output
spec-trace init --update      # refresh generated instruction files
spec-trace init --global      # install the skill into ~/.claude instead

A path filter narrows what is reported, not what is parsed, so a filtered run detects the same orphaned references as a full run.

In CI

- name: Spec check
  run: uvx spec-trace

Configuration

Config is optional. Zero config works for a Python or TypeScript repo keeping specs in docs/specs. Settings live in [tool.spec-trace] in pyproject.toml, or in spec-trace.toml for non-Python repos.

[tool.spec-trace]
specs_dir = "docs/specs"
max_behaviour_length = 100
exclude_dirs = ["node_modules", ".venv", "dist", "build", "__pycache__", ".git"]
exclude_files = ["tests/fixtures_test.py"]   # files whose SPEC: text is data, not a reference

[[tool.spec-trace.languages]]
name = "php"
test_glob = "*Test.php"
comment = "//"
test_function = 'function\s+(test\w+)'

[tool.spec-trace.test_types]   # optional; path prefix -> label in coverage output
"integration-tests/" = "e2e"

Every list setting replaces its default rather than adding to it. Setting exclude_dirs drops the built-in exclusions, so repeat the ones you still want. Defining any language likewise replaces both built-in languages.

exclude_files skips a matching test file entirely, so a real SPEC: reference inside it is ignored too. Reach for it only when a file contains reference-shaped text as data — a better fix is usually to build that text from parts, as tests/spec_fixtures.py does here.

For coding agents

An agent meets this tool at three moments, and each has its own surface:

1. While planning — init installs repo instructions (a Claude Code skill plus an AGENTS.md section) so specs get written with the work, not patched in after a gate failure.
2. On first contact — --help carries a compressed convention: both formats, one example of each, and what the gate fails on.
3. When the gate fails — every ERROR states the fix, and the FAIL line points at spec-trace docs.

The projected files are deliberately thin: the stable format cheat-sheet inline, everything else deferred to spec-trace docs, whose content ships inside the package and versions with it. init --update rewrites the generated region between marker comments, leaving your own text alone.

Known limits

Being honest about what this does not do:

• It checks linkage, not truth. Nothing verifies that a test actually exercises the behaviour its bullet describes. That is delegated to review and agent discipline.
• The test→spec direction is weak at scale. A large codebase adopting this will have thousands of unspecified tests, a number nobody acts on. Treat it as a backlog signal, not a target.
• Parsing is regex, not AST. The "SPEC comment attaches to the next test function" rule is a loose heuristic, and TypeScript describe nesting is discarded. Acceptable for a lint.

Non-goals: no semantic verification, no test running, no coverage measurement, no MCP server, and no runtime dependencies.

Development

uv sync --extra dev
uv run pytest
uv run ruff format . && uv run ruff check .
uv run pyright src/
uv run spec-trace            # the tool gates its own specs

spec-trace uses its own convention: see docs/specs/, with the cross-references in tests/test_spec_linked.py.

Related

maelstrom is a sibling tool for orchestrating parallel agent development. The two share a distribution pattern but have no dependency in either direction: spec-trace is repo-scoped and must run in CI with nothing but Python.

Licence

MIT