spec-probe

Check code against functional requirements in a spec (.md / .docx / .pdf). Use VS Code Copilot chat (/spec-cover, …) or the spec-probe CLI.

Workflow Overview

1. One-Time Setup

Step 1: Install & Skills

Requires Python ≥ 3.11 and pip ≥ 23.0. On Linux/macOS use python3 (not legacy python).

python3 -m pip install -U pip
python3 -m pip install -U spec-probe
spec-probe --version
spec-probe-skill --vscode

• Installs spec-probe and sets up Copilot skills in ~/.copilot/skills/.
• Initializes user directory and .venv under ~/.config/spec-probe/.
• Reload VS Code after running spec-probe-skill.

Optional for Linux: sudo apt install ripgrep (faster grep search).

Step 2: Authenticate & API Keys (.env)

A. GitHub Copilot Token (Required for Verification & Review)

spec-probe-copilot-login

1. Open https://github.com/login/device in your browser.
2. Enter the 8-character code displayed in the terminal.
3. Token is saved automatically to COPILOT_GITHUB_TOKEN in ~/.config/spec-probe/.env.

B. Exacode API Key (Required for High-Volume Jobs)

Exacode is an LGE-internal free LLM gateway. It powers high-frequency pipeline jobs (enrich, search, rescue, report, wiki) without token cost or external rate limits.

1. Open http://exacode-chat.lge.com/key/ (or clear and re-create your key).
2. Copy Key 2 (EXACODE SWE API).
3. Add to ~/.config/spec-probe/.env:

   EXACODE_API_KEY="your_key_2_here"
   EXACODE_BASE_URL="http://exacode-chat.lge.com/v1"
   

(Details: Exacode for Copilot Chat)

Step 3: Configure Target Paths (config.yaml)

Edit ~/.config/spec-probe/config.yaml (use absolute paths):

input:
  module_path: /abs/path/to/your/app
  spec_path: /abs/path/to/spec.docx
  related_modules:
    # paths:
    #   - /abs/path/to/shared/service
    #   - /abs/path/to/core/service

output:
  # Directory to store reports (default: ~/.config/spec-probe/reports)
  # report_dir: /path/to/my/reports

codegraph:
  # Base codebase / monorepo root to index with codegraph (optional)
  # When set, running `spec-probe-codegraph` indexes this root directly.
  # If omitted, defaults to indexing module_path and related_modules.paths.
  # root_path: /abs/path/to/codebase/root

Step 4: Index Code with Codegraph

Builds an AST symbol table and call-graph database essential for accurate C/C++ cross-module symbol resolution:

spec-probe-codegraph

• When codegraph.root_path is set in config.yaml, it indexes the codebase root directly with a single command.
• If not set, it automatically indexes module_path and related_modules.paths.
• You can also pass paths directly: spec-probe-codegraph /path1 /path2. Tests and mocks are excluded automatically. Re-running syncs changes.

Step 5: Draft Domain Pack (Run once per module)

In VS Code Copilot chat:

1. Select Model in Chat Picker: Choose Gemini 3.8 Flash (recommended for huge 265k context window and fast code analysis) or GPT-6 Luna.
2. Run slash command:

   /spec-pack
   

Why run via Copilot Chat Agent instead of internal API calls? Domain Pack creation is an agentic, multi-step synthesis: it scans AST/symbols, reads 3–8 core architecture and state-machine files, cross-checks spec vocabulary, drafts project-specific rules, and validates patterns with spec_pack.py validate --strict. Running as an interactive Copilot Chat agent lets the developer review and tailor domain heuristics with human-in-the-loop flexibility.

Step 6: Parse Spec (Run once per spec)

In VS Code Copilot chat:

/spec-cover parse spec summary

Parses the requirements document into a structured catalog cached under ~/.config/spec-probe/reports/specs/ so subsequent verifications are fast.

Zero Token Cost by Default (0 LLM Tokens): Spec parsing is 100% deterministic local Python execution (using regex, AST, and table parsing via load_spec_file). No external LLM API is called, consuming 0 tokens. In Copilot Chat, any chat model picker works equally well (e.g. lightweight models like Haiku or Flash).

(Novel / Unstructured Spec Fallback): If a specification lacks standard tables and numbered headers, spec-probe parse --spec <path> --llm-fallback uses per-section extraction (llm.jobs.parse) to accurately capture requirements without context overflows.

2. Daily Workflow (VS Code Copilot Chat)

Once one-time setup is complete, use chat for regular verification:

Step 1: Verify Requirements (/spec-cover)

• Verify a specific requirement:

  /spec-cover verify FR-001
  

• Or verify by requirement text:

  /spec-cover check requirement: The module shall initialize within 500ms
  

• Or verify a section:

  /spec-cover verify section 3.2
  

(Note: In Copilot chat, typing naturally without /spec-cover like verify FR-001 also works via auto-intent matching, but /spec-cover ... is recommended for explicit routing).

During verification, results and a temporary draft report are automatically persisted to ~/.config/spec-probe/cache/ (last_verify_session.json and draft_verify_latest.md). No unwanted report files are written to your project output directory yet.

Step 2: Audit Report (/spec-audit) — Mandatory Quality Gate

Crucial & Required Step: Automated scan results from Step 1 are treated as unconfirmed hypotheses. This step audits the temporary draft results, eliminates AI hallucinations, validates code citations line-by-line, and confirms true implementation gaps with zero risk of false positives.

(Note on token cost: /spec-audit is extremely token-efficient — it does not read the entire codebase. It runs targeted ripgrep searches only on suspected gaps and NOT_FOUND items, consuming very few tokens while boosting accuracy to 100%).

In VS Code Copilot chat:

1. Select Model in Chat Picker: Choose Gemini 3.8 Flash (recommended for deep reasoning and whole-file code context) or GPT-6 Luna.
2. Run slash command:

   /spec-audit
   

Why run via Copilot Chat Agent with a strong model? Like /spec-pack, /spec-audit is an interactive agentic review. It treats tool verdicts and "missing" bullets as hypotheses, actively reads surrounding source code, traces evidence chains across caller-callee graphs, resolves ambiguous symbols, confirms true implementation gaps, and reports findings directly in the chat.

Step 3: Export Final Report File

Once you have reviewed the audited findings in chat, export the official report:

/spec-cover write report

Generates the final verified .md and standalone .html reports under your configured report directory (e.g. reports/verifications/ or path configured in config.yaml). Because this is run after auditing, your published reports will always represent the highest-accuracy ground truth.

3. Technical Tuning & LLM Jobs (config.json)

config.json is dedicated to technical pipeline tuning and slotting LLM models for each phase.

LLM Pipeline Phases (llm.jobs)

config.json.example sets verify / final_review → gemini-3.8-flash, other jobs → gpt-6-luna (after spec-probe-copilot-login). Re-run spec-probe install --init (or skill install) to sync llm.jobs into ~/.config/spec-probe/config.json. Switch high-volume jobs to internal Exacode:

"enrich": { "provider": "exacode", "model": "Chat-EXACODE-A" }

Key Pipeline Parameters (pipeline)

• parallel_workers: Number of concurrent workers for batch verification (default: 1 or 4).
• context_lines: Number of source code context lines included around grep matches (default: 40).
• batch_size: Batch size when processing large specifications (default: 20).
• final_review_enabled: Enables the secondary whole-file verification pass (default: true).

4. Updating spec-probe

When a new version is released, updating is a single command:

spec-probe update

• Automatically upgrades the package via pip (or git pull if running from a local dev clone).
• Refreshes IDE skills and slash commands in ~/.copilot/skills/ and ~/.cursor/skills/.
• Preserves 100% of your configurations: never overwrites config.yaml, config.json, or .env.
• To check if an update is available without installing:

  spec-probe update --check-only
  

CLI (optional)

spec-probe verify --module /abs/app --spec /abs/spec.md --fr FR-001
spec-probe check "The module shall …"
spec-probe parse --spec /abs/spec.md --summary

Paths default from ~/.config/spec-probe/config.yaml when omitted.