spec-probe

Check code against functional requirements in a spec (.md / .docx / .pdf). Use VS Code Copilot chat (/spec-cover, …) or the spec-probe CLI.

Workflow Overview

1. One-Time Setup

Step 1: Install & Skills

pip install spec-probe
spec-probe-skill --vscode

• Installs spec-probe and sets up Copilot skills in ~/.copilot/skills/.
• Initializes user directory and .venv under ~/.config/spec-probe/.
• Reload VS Code after running spec-probe-skill.

Optional for Linux: sudo apt install ripgrep (faster grep search).

Step 2: Authenticate & API Keys (.env)

A. GitHub Copilot Token (Required for Verification & Review)

spec-probe-copilot-login

1. Open https://github.com/login/device in your browser.
2. Enter the 8-character code displayed in the terminal.
3. Token is saved automatically to COPILOT_GITHUB_TOKEN in ~/.config/spec-probe/.env.

B. Exacode API Key (Required for High-Volume Jobs)

Exacode is an LGE-internal free LLM gateway. It powers high-frequency pipeline jobs (enrich, search, rescue, report, wiki) without token cost or external rate limits.

1. Open http://exacode-chat.lge.com/key/ (or clear and re-create your key).
2. Copy Key 2 (EXACODE SWE API).
3. Add to ~/.config/spec-probe/.env:

   EXACODE_API_KEY="your_key_2_here"
   EXACODE_BASE_URL="http://exacode-chat.lge.com/v1"
   

(Details: Exacode for Copilot Chat)

Step 3: Configure Target Paths (config.yaml)

Edit ~/.config/spec-probe/config.yaml (use absolute paths):

input:
  module_path: /abs/path/to/your/app
  spec_path: /abs/path/to/spec.docx
  related_modules:
    # paths:
    #   - /abs/path/to/shared/service
    #   - /abs/path/to/core/service

output:
  # Directory to store reports (default: ~/.config/spec-probe/reports)
  # report_dir: /path/to/my/reports

codegraph:
  # Base codebase / monorepo root to index with codegraph (optional)
  # When set, running `spec-probe-codegraph` indexes this root directly.
  # If omitted, defaults to indexing module_path and related_modules.paths.
  # root_path: /abs/path/to/codebase/root

Step 4: Index Code with Codegraph

Builds an AST symbol table and call-graph database essential for accurate C/C++ cross-module symbol resolution:

spec-probe-codegraph

• When codegraph.root_path is set in config.yaml, it indexes the codebase root directly with a single command.
• If not set, it automatically indexes module_path and related_modules.paths.
• You can also pass paths directly: spec-probe-codegraph /path1 /path2. Tests and mocks are excluded automatically. Re-running syncs changes.

Step 5: Draft Domain Pack (Run once per module)

In VS Code Copilot chat:

1. Select Model in Chat Picker: Choose Gemini 3.8 Flash (recommended for huge 265k context window and fast code analysis) or GPT-6 Luna.
2. Run slash command:

   /spec-pack
   

Why run via Copilot Chat Agent instead of internal API calls? Domain Pack creation is an agentic, multi-step synthesis: it scans AST/symbols, reads 3–8 core architecture and state-machine files, cross-checks spec vocabulary, drafts project-specific rules, and validates patterns with spec_pack.py validate --strict. Running as an interactive Copilot Chat agent lets the developer review and tailor domain heuristics with human-in-the-loop flexibility.

Step 6: Parse Spec (Run once per spec)

In VS Code Copilot chat:

/spec-cover parse spec summary

Parses the requirements document into a structured catalog cached under ~/.config/spec-probe/reports/specs/ so subsequent verifications are fast.

Zero Token Cost by Default (0 LLM Tokens): Spec parsing is 100% deterministic local Python execution (using regex, AST, and table parsing via load_spec_file). No external LLM API is called, consuming 0 tokens. In Copilot Chat, any chat model picker works equally well (e.g. lightweight models like Haiku or Flash).

(Novel / Unstructured Spec Fallback): If a specification lacks standard tables and numbered headers, spec-probe parse --spec <path> --llm-fallback uses per-section extraction (llm.jobs.parse) to accurately capture requirements without context overflows.

2. Daily Workflow (VS Code Copilot Chat)

Once one-time setup is complete, use chat for regular verification:

Step 1: Verify Requirements (/spec-cover)

• Verify a specific requirement:

  /spec-cover verify FR-001
  

• Or verify by requirement text:

  /spec-cover check requirement: The module shall initialize within 500ms
  

• Or verify a section:

  /spec-cover verify section 3.2
  

(Note: In Copilot chat, typing naturally without /spec-cover like verify FR-001 also works via auto-intent matching, but /spec-cover ... is recommended for explicit routing).

During verification, results and a temporary draft report are automatically persisted to ~/.config/spec-probe/cache/ (last_verify_session.json and draft_verify_latest.md). No unwanted report files are written to your project output directory yet.

Step 2: Audit Report (/spec-audit) — Mandatory Quality Gate

Crucial & Required Step: Automated scan results from Step 1 are treated as unconfirmed hypotheses. This step audits the temporary draft results, eliminates AI hallucinations, validates code citations line-by-line, and confirms true implementation gaps with zero risk of false positives.

(Note on token cost: /spec-audit is extremely token-efficient — it does not read the entire codebase. It runs targeted ripgrep searches only on suspected gaps and NOT_FOUND items, consuming very few tokens while boosting accuracy to 100%).

In VS Code Copilot chat:

1. Select Model in Chat Picker: Choose Gemini 3.8 Flash (recommended for deep reasoning and whole-file code context) or GPT-6 Luna.
2. Run slash command:

   /spec-audit
   

Why run via Copilot Chat Agent with a strong model? Like /spec-pack, /spec-audit is an interactive agentic review. It treats tool verdicts and "missing" bullets as hypotheses, actively reads surrounding source code, traces evidence chains across caller-callee graphs, resolves ambiguous symbols, confirms true implementation gaps, and reports findings directly in the chat.

Step 3: Export Final Report File

Once you have reviewed the audited findings in chat, export the official report:

/spec-cover write report

Generates the final verified .md and standalone .html reports under your configured report directory (e.g. reports/verifications/ or path configured in config.yaml). Because this is run after auditing, your published reports will always represent the highest-accuracy ground truth.

3. Technical Tuning & LLM Jobs (config.json)

config.json is dedicated to technical pipeline tuning and slotting LLM models for each phase.

LLM Pipeline Phases (llm.jobs)

The hybrid strategy balances high reasoning quality for critical verification with free high-throughput inference for pre/post-processing:

Key Pipeline Parameters (pipeline)

• parallel_workers: Number of concurrent workers for batch verification (default: 1 or 4).
• context_lines: Number of source code context lines included around grep matches (default: 40).
• batch_size: Batch size when processing large specifications (default: 20).
• final_review_enabled: Enables the secondary whole-file verification pass (default: true).

4. Updating spec-probe

When a new version is released, updating is a single command:

spec-probe update

• Automatically upgrades the package via pip (or git pull if running from a local dev clone).
• Refreshes IDE skills and slash commands in ~/.copilot/skills/ and ~/.cursor/skills/.
• Preserves 100% of your configurations: never overwrites config.yaml, config.json, or .env.
• To check if an update is available without installing:

  spec-probe update --check-only
  

CLI (optional)

spec-probe verify --module /abs/app --spec /abs/spec.md --fr FR-001
spec-probe check "The module shall …"
spec-probe parse --spec /abs/spec.md --summary

Paths default from ~/.config/spec-probe/config.yaml when omitted.