SpecTracer

A CLI tool that takes your Gherkin .feature files as the source of truth for what needs testing, then collates test results from your Unit, Integration, and E2E suites into a single, self-contained HTML report — plus an optional machine-readable JSON twin for CI automation.

Feature files define the scope. Tags on scenarios link them to test results across layers. The report shows:

• What percentage of scenarios actually have test completion (the headline metric).
• Where that coverage exists across layers (per-scenario pass/fail/skip breakdown).
• The overall test pyramid — test count, duration, and pass rate per layer.
• Every failure's stack trace, in one place.

An optional JSON report (conforming to spectracer-report.schema.json) mirrors the same data for scripting — PR bots, custom gating, dashboards — without scraping HTML.

The tool is tech-stack agnostic: it only needs Gherkin .feature files, JUnit XML, and Cucumber JSON, so it works regardless of what languages or frameworks produced them.

Why

• Fragmented visibility — unit, integration, and E2E tests usually live in different directories or repos, with no single view of overall coverage.
• Inverted pyramids — teams unknowingly accumulate slow E2E tests instead of fast unit tests, and don't notice until CI is painfully slow.
• No traceability — it's hard to know if a specific business scenario is actually complete across all the layers it should.
• Tooling lock-in — most reporting tools are tied to one framework (Allure for Java, Cypress Dashboard for Cypress). This one isn't.

Installation

Requires Python 3.12+.

pip install spec-tracer

Or with uv:

uv pip install spec-tracer

Once installed, the spec-tracer CLI is available globally.

From source

git clone https://github.com/ampyard/spec-tracer.git
cd spec-tracer
uv sync

To build a wheel for local testing:

uv build

The wheel is written to dist/spec_tracer-*.whl. Install it with uv pip install dist/*.whl.

To run this repo's own test suite and regenerate its dogfooded report (reports/spectracer-report.html), matching what CI does:

uv sync --group dev
uv run python run_local.py

Quick Start

1. Write .feature files describing your scenarios, tagged so test results can link back to them (see Tagging Convention below).

2. Run your test suites and produce JUnit XML (unit/integration) and/or Cucumber JSON (E2E) output.

3. Create a spectracer.config.json in your project root:

   {
     "features": ["./features"],
     "unit": { "": ["./reports/unit.xml"] },
     "integration": { "": ["./reports/integration.xml"] },
     "e2e": { "": ["./reports/e2e.json"] },
     "output": "./report.html"
   }
   

4. Run the tool:

   uv run spec-tracer
   

   It auto-discovers spectracer.config.json in the current directory — no flags needed. Open the generated report.html in a browser.

To point at a config file with a different name or location, pass it as the only argument:

uv run spec-tracer path/to/other-config.json

Note on shells: use forward slashes (./spectracer.config.json) or a bare filename. A leading .\ (PowerShell-style) can be mangled by POSIX-style shells (Git Bash, WSL), since backslash is their escape character there.

Tagging Convention

Feature files and test results connect via paired identity tags (@id: on scenarios, @scenario: on results) and layer requirement tags (@require-*). There are three tag roles:

Feature: User Login

  @id:FC-42 @regression @require-unit:auth @require-integration:auth @require-e2e:auth
  Scenario: Successful login with valid credentials
    Given the user is on the login page
    When they enter valid credentials
    Then they should be redirected to the dashboard

  @id:FC-43
  Scenario: Login with invalid password shows error
    Given the user is on the login page
    When they enter an invalid password
    Then an error message should be displayed

• Scenario identity tags (@id:FC-42) — placed on Gherkin scenarios. Declares the scenario's stable identity.
• Test reference tags (@scenario:FC-42) — placed on test results (via pytest markers, test names, or annotations). A result with @scenario:FC-42 links to any scenario carrying @id:FC-42.
• Layer requirement tags (@require-unit, @require-integration, @require-e2e) — declare which layers must have coverage for this scenario. These are never used for linking, and the tool flags any declared layer that ends up with zero linked results.

Module-scoped requirements

@require-unit, @require-integration, and @require-e2e accept an optional :modulename suffix, e.g. @require-unit:auth or @require-e2e:checkout. This pairs with module-keyed entries in the config file's unit/integration/e2e objects (see below) — a module-scoped requirement is only satisfied by a result registered under that exact module. An unscoped result (config key "") never satisfies it, and a bare @require-unit / @require-e2e (no module) is satisfied by any linked result for that layer regardless of module.

Matching rules

• Prefix-filtered — only @scenario:VALUE on results and @id:VALUE on scenarios participate. @regression, @smoke, and other tags are ignored during linking.
• Exact VALUE match — @scenario:FC-42 matches @id:FC-42 only, not @id:FC-4 or @id:FC-42-smoke.
• OR logic within prefixed tags — a result tagged [@scenario:FC-42, @scenario:FC-43] links to any scenario carrying either @id:FC-42 or @id:FC-43.
• Scenario tags only — tags on the Feature: line are not inherited by scenarios.
• @require-* tags are excluded from linking — no collision with linking tags is possible.
• Tag collisions link everywhere — if two scenarios (in the same or different feature files) share an @id: value, one matching test result links to both.

Where the tool looks for tags in test results

• JUnit XML (unit/integration): the name attribute, classname attribute, or <properties><property> elements — whichever your framework populates.
• Cucumber JSON (E2E): the native scenario-level tags array.

Configuration File

The tool is configured entirely through a JSON file — there are no CLI flags. Default filename is spectracer.config.json at the project root; pass an explicit path as the sole CLI argument to use a different one.

{
  "features": ["./features"],
  "unit": {
    "": ["./reports/unit.xml"],
    "billing": ["./reports/billing-unit.xml"]
  },
  "integration": {
    "": ["./reports/integration.xml"]
  },
  "e2e": {
    "": ["./reports/e2e.json"],
    "checkout": ["./reports/checkout-e2e.json"]
  },
  "output": "./report.html",
  "output_json": "./report.json",
  "error_on_failure": false,
  "fail_on": ["pyramid", "e2e_runtime"],
  "health_checks": {
    "progress_threshold_green": 80,
    "progress_threshold_amber": 50,
    "e2e_duration_amber_seconds": 600,
    "e2e_duration_red_seconds": 1800
  }
}

The Report

The generated HTML is a single self-contained file (all CSS/JS inlined — a monospace font loads from a CDN with a system fallback — safe to email or archive) with five sections:

1. Coverage Progress Summary — headline stats for declared-tests matched and scenarios fully matched, plus the four health checks. Color-coded green/amber/red using the configurable thresholds.
2. Global Pyramid Dashboard — a 3-tier visualization (E2E / Integration / Unit) with test counts, duration, and pass rate per layer, plus health indicators for an inverted pyramid or an E2E layer with excessive runtime.
3. Feature Traceability & Scenario Matrix — a searchable, expandable tree: Feature → Scenario → Layer results, with full Gherkin text, declared layer requirements (✓/✗), and per-test pass/fail/skip status. (Failure stack traces live on the Failure Breakdown page.)
4. Detailed Failure Breakdown — every failed test across all layers, with feature/scenario context and full stack trace on expand.
5. Unlinked Tests — test results whose tags didn't match any scenario, to help catch orphaned or mis-tagged tests.

Machine-Readable JSON Report

Setting output_json in the config produces a JSON file alongside the HTML report, built from the exact same internal data — the two outputs can never drift apart. It conforms to spectracer-report.schema.json (Draft 7), which is the authoritative contract; the highlights:

• summary.completion / summary.pyramid / summary.health — the headline stats, per-layer stats, and health status (green/amber/red with reasons[]) shown on the HTML dashboard. Note that summary.completion.percent is the declared tests matched percentage (satisfied/required); tested/total are scenario counts and percent is not tested / total.
• features[].scenarios[].results[] — every linked test result per scenario, with module, duration (milliseconds) and failureMessage omitted rather than null when not available, and layer requirement satisfaction under requirements[].
• unlinkedTests[] — the same orphaned results shown in the HTML report's "Unlinked Tests" page.
• config — a verbatim echo of the resolved config used to produce the report, for provenance if the JSON is archived independently of the repo.

{
  "features": ["./features"],
  "unit": { "": ["./reports/unit.xml"] },
  "e2e": { "": ["./reports/e2e.json"] },
  "output": "./report.html",
  "output_json": "./report.json"
}

Useful for PR bots, custom CI gating beyond error_on_failure, or feeding coverage numbers into a dashboard — without scraping the HTML.

Historical Trends

output_json already emits summary.completion / summary.pyramid / summary.health on every run — point your CI at whatever metrics system your team already has and post that block after each report is generated:

- name: Generate report
  run: spec-tracer

- name: Post coverage metrics
  run: |
    curl -X POST "$METRICS_ENDPOINT" \
      -H "Content-Type: application/json" \
      -d @reports/spectracer-report.json

That's the whole integration — one curl (or your metrics SDK's equivalent) reading a file that already exists. Whatever's on the receiving end (Grafana, Datadog, an internal dashboard, even a spreadsheet webhook) owns the history, the retention policy, and the charting. See the CI/CD guide for a worked example.

Behavior Reference

What This Tool Doesn't Do

• Not a test runner — it only parses results after your tests have already run.
• Not a source-code parser — it never reads your .java/.py/.js files, only .feature files and test-result output.
• No tag expressions — matching is exact string equality only; no not/and/or boolean tag logic.