skillscan-trace

Behavioral execution engine for MCP-based AI agent skills.

skillscan-trace runs a skill against a real language model inside an instrumented, isolated environment and records everything the model does: every file it reads, every network request it makes, every environment variable it accesses, every binary it probes. The output is a structured, machine-readable trace report.

It is the dynamic analysis counterpart to skillscan, which performs static analysis. Together they form two legs of the skillscan family:

skillscan family
├── skillscan        — static analysis (pattern matching, ML classifier)
├── skillscan-trace  — behavioral execution engine  ← this repo
└── skillscan-lint   — schema and format validation (planned)

What it does

A skill is a Markdown file that becomes a system prompt for an AI agent. Most skills are benign. Some are malicious — they instruct the agent to exfiltrate credentials, probe the filesystem, call attacker-controlled servers, or hijack the agent's behavior through prompt injection embedded in tool output.

Static analysis catches the obvious cases. Behavioral analysis catches the rest: conditional payloads that only activate in certain environments, obfuscated instructions that decode at runtime, and prompt injection delivered through external content the skill fetches.

skillscan-trace works by:

1. Loading the skill's SKILL.md as the system prompt for a local language model
2. Sending a realistic user prompt that exercises the skill's stated functionality
3. Driving the model's tool-use loop through an instrumented MCP server that intercepts every tool call
4. Checking each call against a canary taxonomy (credential files, wallet paths, ENV vars, binary probing, network destinations)
5. Emitting a structured trace report (JSON + SARIF) with every observed behavior and any findings

The model runs locally via Ollama — no API key required, no cloud dependency. API providers (OpenAI, OpenRouter) are supported for users who prefer them or want access to more capable models.

Status

v0.2.0 — verdict banner + full CI matrix. All phases implemented and 199/199 tests pass across Python 3.11, 3.12, and 3.13. The tool is installable and usable today.

Quick start

# Install from source
git clone https://github.com/kurtpayne/skillscan-trace
cd skillscan-trace
pip install -e .

# Run a trace with OpenAI
export OPENAI_API_KEY=sk-...
skillscan-trace run ./path/to/skill/

# Run with OpenRouter (200+ models via one key)
export OPENROUTER_API_KEY=sk-or-...
skillscan-trace run ./skill/ --provider openrouter --model mistralai/mistral-7b-instruct

# Run with a local Ollama model (no API key required)
# Model must support tool calling — llama3.1:8b and llama3.2:3b are verified
skillscan-trace run ./skill/ --provider ollama --model llama3.1:8b

# Dry run — validate config and skill loading without calling the LLM
skillscan-trace run ./skill/ --dry-run

# Run with explicit base URL (Azure, Mistral, etc.)
skillscan-trace run ./skill/ --base-url https://api.mistral.ai/v1 --api-key $MISTRAL_KEY

# Output formats
skillscan-trace run ./skill/ --format sarif   # SARIF 2.1.0 for CI
skillscan-trace run ./skill/ --format json    # native trace format
skillscan-trace run ./skill/ --format text    # human-readable summary

# Verify connectivity
skillscan-trace check
skillscan-trace check --provider openrouter
skillscan-trace check --provider ollama

Skill format support

skillscan-trace handles all skill formats found in the wild:

Docker

A single image supports both run (single trace) and serve (HTTP API) modes. The same image powers the hosted service at trace.skillscan.sh and enterprise self-hosting.

Pull the image:

docker pull kurtpayne/skillscan-trace:latest

Run mode (single trace)

Mount your skill and pass an API key via env var. The container exits after the trace completes.

docker run --rm \
  -v "$(pwd)":/data \
  -e OPENAI_API_KEY=sk-... \
  kurtpayne/skillscan-trace run /data/SKILL.md

Use OpenRouter instead:

docker run --rm \
  -v "$(pwd)":/data \
  -e OPENROUTER_API_KEY=sk-or-... \
  kurtpayne/skillscan-trace run /data/SKILL.md --provider openrouter

Fully local with Ollama on the host:

docker run --rm --network host \
  -v "$(pwd)":/data \
  kurtpayne/skillscan-trace run /data/SKILL.md --provider ollama

Serve mode (HTTP API)

Starts the FastAPI server on port 8080. Callers provide their own API key per request (BYOK) — keys are never logged or stored.

docker run -d -p 8080:8080 --name sst \
  kurtpayne/skillscan-trace serve

curl http://localhost:8080/v1/health

For persistent caching across restarts, mount volumes for /trace-cache and /trace-output. A full docker-compose.yml is provided in the repository root.

Output: Trace Report

Every trace produces a JSON trace report and optionally a SARIF report.

{
  "schema_version": "1.0.0",
  "trace_id": "trc_20260320_abc123",
  "skill": {
    "path": "./my-skill/SKILL.md",
    "name": "git-helper",
    "sha256": "a1b2c3..."
  },
  "model": {
    "provider": "ollama",
    "model": "llama3.1:8b",
    "version": "..."
  },
  "prompt": "Help me commit my changes",
  "duration_ms": 4823,
  "tool_calls": [...],
  "findings": [...],
  "summary": {
    "total_tool_calls": 7,
    "finding_count": 1,
    "severity_max": "HIGH",
    "clean": false
  }
}

Relationship to skillscan-security

skillscan-trace is a sibling project to skillscan-security. The two projects share:

• Finding schema: The same finding IDs (EXF-001, MAL-001, IOC-001, etc.) and severity levels
• Canary taxonomy: The same list of high-value target paths and ENV var names
• Domain allowlist: trace/domains/verified.yml from skillscan-security is the source of truth; skillscan-trace consumes it
• Corpus feedback loop: Traces that produce findings can be reviewed and added to the skillscan-security corpus as sandbox_verified/ examples, improving the ML classifier's recall on behavioral patterns that static analysis misses

Privacy & Security

Your API keys are never stored, logged, or transmitted by SkillScan — they are passed directly to the LLM provider you chose over HTTPS and held in memory only for the duration of the run.

• No telemetry, no analytics, no phone-home — skillscan-trace makes zero network requests beyond the LLM API calls you explicitly authorize
• Local-first — the canary server runs in-process on your machine; nothing leaves your network except LLM API calls
• Hosted service (trace.skillscan.sh) — follows the BYOK model; your API key is forwarded to the LLM provider and never stored or cached; only the trace report is cached
• No user identity — no accounts, no login, no tracking
• Ollama — fully local, zero external network requests, no API key required

See PRIVACY.md for the full data flow diagram and detailed explanation.

Contributing

See CONTRIBUTING.md.

License

Apache-2.0