spec-eval 0.2.0

A trust-eval for your specs: coverage, drift, and sufficiency of markdown specs against code.

spec-eval — Build fast with AI. Validate with specs.

Your code changes fast. Your docs fall behind. spec-eval keeps them honest.

The trust layer for spec-driven development — iterate fast, keep the spec and code in agreement.

A trust-eval for your specs — coverage, drift, and sufficiency, measured against the code.

Get started · Tutorial · No API key? · Second opinion · FAQ

A spec is short for specification — a plain-English description of what your code should do. spec-eval keeps one beside your code — per file, or per folder — and checks the two still agree, so the spec stays worth trusting. How far you lean on it is up to you. No specs yet? spec-eval writes them for you.

Table of contents

• The problem
• What it checks
• What it creates
• Try it
  • Run via prompt chat (no setup)
  • Run in the terminal, no key (Claude subscription)
  • Run in the terminal with an API key (for CI)
• How the scores are made
• Reading the scores
• Higher stakes? Get a second opinion
• Proof it works
• More
• FAQ
• Feedback
• Acknowledgements

The problem

Specs go stale the moment code moves on — and one stale spec is enough to stop a reader trusting all of them. AI coding agents make code move faster than any hand-maintained doc can follow.

spec-eval treats specs like code: run a check — via a chat prompt, an ordinary command, or a CI step — and the spec is measured against the source. A spec you can check is a spec you can trust.

What it checks

Three simple questions:

Check	The question it answers	How
coverage	Does this code have a spec at all?	Just counts files. Free — no AI, no key.
drift	Does the spec say something the code doesn't do?	An AI reads each spec and its code and lists the clashes.
sufficiency	How completely does the spec capture the code's behavior?	An AI lists what the spec misses and scores it 0–1 — an indicator, not a guarantee.

Your code is the source of truth — the spec is what gets graded against it.

What it creates

Everything it writes is plain markdown, right next to your code:

 your code                 specs beside it              spec-reports/
┌────────────┐             ┌────────────┐             ┌────────────────┐
│ parser.py  │  generate   │ parser.md  │  coverage   │ coverage.md    │
│ api.py     │ ──────────► │ api.md     │  audit      │ report.md      │
│ …          │             │ …          │ ──────────► │ sufficiency.md │
└────────────┘             └────────────┘ sufficiency └───────┬────────┘
                                                              │
                                                              ▼
                                                       SPEC-HEALTH.md
                                                    (one-page scorecard)

Try it

Prefer a guided walkthrough? Getting-started tutorial →

The commands — in the order you'll typically use them:

• coverage — which files have no spec yet? (free — no AI)
• generate — writes a plain-English spec beside each file that has none (uses AI)
• sufficiency — how completely does each spec capture the code? Scored 0–1 (uses AI)
• audit — the drift check: does any spec clash with the code? Run it at every milestone (uses AI)

None of them change your files — reports go to spec-reports/. The exception is generate, which writes new specs beside your code: review them and commit the keepers like code.

The (uses AI) commands (generate, sufficiency, audit) need an AI behind them — coverage never does. Three ways to run them, least setup first:

Run via prompt chat (no setup)

Ask the coding agent you already have (Claude Code, Copilot, Cursor, …) — nothing to install, no key, answers land in the chat:

Read path/to/spec-eval/skills/spec-check/SKILL.md and follow it —
first check coverage, then check my specs against my code.

The path is wherever you cloned this repo — no clone? Give your agent the file's link.

Optionally save reports — add:

and save the results to spec-reports/

The agent writes them itself (its own scores; the terminal adds the exact coverage % and a run history).

Make it a standing command (optional, one-time) — copy the skills/ into your agent's skills folder (Claude Code: .claude/skills/). Then the prompt is the command:

mkdir -p .claude/skills && cp -r path/to/spec-eval/skills/* .claude/skills/

Which files have no spec yet?        # coverage
Write a spec for src/parser.py       # generate
Check my specs against my code       # sufficiency + audit

Run in the terminal, no key (Claude subscription)

Install once, then add --model claude-code to any (uses AI) command — spec-eval routes through the claude CLI you're already logged into (no key; your ANTHROPIC_API_KEY is hidden from the call, so it can't accidentally bill the paid API). The same four commands, in the same order as the list above:

pip install spec-eval                                     # or, from a clone:  pip install -e .
spec-eval coverage    ./your-project                      # free — no AI, no key
spec-eval generate    ./your-project --model claude-code
spec-eval sufficiency ./your-project --model claude-code
spec-eval audit       ./your-project --model claude-code

Every terminal run writes its reports to spec-reports/ (plus a runs.jsonl history line).

Run in the terminal with an API key (for CI)

Export a key and run any command above without --model claude-code — an API key is the default. This is the option for CI, the checks that run automatically on every push (store the key as a repo secret):

export ANTHROPIC_API_KEY=sk-ant-...     # or OPENAI_API_KEY / GOOGLE_API_KEY

[!NOTE] A typical first run is coverage → generate → sufficiency, then at every milestone after: audit — plus sufficiency when you've added behavior. The tutorial has step-by-step guides for starting with no specs and for existing docs.

[!TIP] Version control is also your preview. The three checks never modify your project; generate adds ordinary new files — review them with git diff, edit what's off, and drop any you don't want with git checkout -- <file>. (A full generate --overwrite rewrites everything, so save it for first setup.)

A few things worth knowing:

• No config needed — by default each spec pairs with the code file of the same name beside it (parser.md ↔ parser.py). Prefer one spec per folder? --layout per-dir (FAQ).
• Any language, three providers — swap the model with --model openai:…, google:…, or claude-code (Anthropic is the default); code is read as plain text. Want another provider? Open an issue.
• Your other docs are safe — by default only a .md next to code with the same name counts as a spec, so your READMEs are left alone. (Folder specs and overview indexes only appear if you opt in with --layout / --overview.)
• What's shared — to grade a spec, its code and text go to your AI provider. That's it. (coverage sends nothing — it runs fully on your machine.)

Keep it honest on every commit — add one line to a git hook (a script git runs before each commit) so new code always needs a spec; copy-paste setup in the tutorial:

spec-eval coverage . --min 90    # fails the commit if coverage drops below 90%  (free — no AI)

How the scores are made

Three scores, three simple ideas:

• coverage = spec files you have ÷ spec files you need → a percent. Just counting, no AI. (9 of 10 code files have a spec → 90%.)
• drift = the number of spots where the spec and the code flat-out disagree. 0 = they match.
• sufficiency = how much of what the code does is actually written in the spec, from 0 to 1. (1.0 = it's all there; 0.1 = almost none of it is.)

Those three are per spec. The repo's one sufficiency score is just their average — each spec's 0–1, added up and divided by how many specs were scored. (coverage is already one repo-wide %; drift is the total count across specs.)

coverage is pure counting. The other two need an AI reader: it reads each spec next to its code and grades the pair against a fixed rubric — one model call per pair. Two things follow from that:

[!NOTE] AI scores wobble: the same spec might get 0.78 one run and 0.72 the next, like two teachers grading the same essay. Watch the trend, not the decimal — a jump like 0.72 → 0.85 is a real improvement.

[!IMPORTANT] AI calls spend tokens — through your API key, your Claude subscription with --model claude-code, or your coding agent's own usage if you asked in chat. Every CLI run prints its exact call and token counts.

Want the exact rules behind each score? spec-eval specs its own scoring — read spec_eval/coverage.md (how coverage counts), spec_eval/rubric.md (the drift rules), and spec_eval/sufficiency.md (how the 0–1 score is decided).

Reading the scores

Each run writes a short report to spec-reports/ (a .md you read, a .json for tools):

• coverage — the % of your code that has a spec, and which files don't. (live example: coverage.md)
• drift — each place a spec and the code disagree, with a suggested fix. 0 is clean. (live example: report.md)
• sufficiency — a 0–1 score per spec (worst first), listing what's missing with a searchable code pointer (file.py (function)). 1.0 = the grader found nothing missing — guidance, not a guarantee. (live example: sufficiency.md)

The three examples are spec-eval grading itself, rolled up in SPEC-HEALTH.md.

Every command also appends a line to runs.jsonl — timestamp, git commit, scores — so you can track change over time.

Higher stakes? Get a second opinion

drift and sufficiency are AI judgments — one model can miss something or be too generous. For a spec that needs extra attention — a critical path, complex design, or compliance code — run the same check with two different AI vendors and compare. Give each run its own --out folder (otherwise the second run overwrites the first report):

spec-eval audit ./your-project --model anthropic:claude-opus-4-8 --out spec-reports/claude
spec-eval audit ./your-project --model google:gemini-3.5-flash   --out spec-reports/gemini

diff spec-reports/claude/report.md spec-reports/gemini/report.md   # or read them side by side

How to read two reports:

• Both flag it → almost certainly real. Fix these first.
• Only one flags it → a judgment call. Read the quoted evidence and decide.
• Sufficiency scores differ a little → normal wobble. Compare the gaps lists, not the decimals — the same missing behavior named by both graders is the signal.

Each vendor reads its own key from the environment — ANTHROPIC_API_KEY, OPENAI_API_KEY, GOOGLE_API_KEY — so put both keys in one .env and each run picks up the one it needs. (Want one fewer key? Swap either side for --model claude-code, which uses your Claude Code login instead.)

Proof it works

• spec-eval specs itself — every module in spec_eval/ has its own spec: coverage 100% · drift 0 · sufficiency ≈0.86 (receipt — the exact, dated score).

More

• Works with spec-driven development. SDD's goal is a spec you can trust, and spec-eval is that trust layer. Use it three ways: grade the specs you already have, generate specs for code that has none, or check as you write (with the skills). Spec-first, code-first, or in between — it meets you where you are.
• Grow the spec and the code together. The best version of "in between": draft a little spec, write a little code, check they still agree — repeat. The two can even move in parallel (you shape the spec while your coding agent writes the code; the check is where they meet), so the spec is born accurate instead of written up afterward.
• Just want to vibe code? Fine — build first, check later, automatically: a pre-commit gate and a GitHub Action can run the checks for you — see Make it routine.
• Higher-stakes spec? Run it past two vendors and compare — see Higher stakes? Get a second opinion above.

FAQ

Common questions — from "which command do I run first?" through costs, layouts, and score accuracy — live in FAQ.md.

Feedback

Found a bug, a bad score, or a spec that misses the point? Open an issue — a report snippet and the command you ran is plenty.

Acknowledgements

Built on the Anthropic, OpenAI, and Google SDKs (MIT / Apache-2.0) + PyYAML.

---

Terms you may be searching for: software specification · documentation drift · living documentation · docs as code · backfill documentation · spec-driven development (SDD) · agentic development · AI-assisted coding · vibe coding · vibe engineering · AI code documentation · LLM-as-judge / trust eval. MIT licensed.