spooled-ai 0.9.0

CI for AI agents - behavioral fingerprinting and drift detection

Spooled — Behavioral CI for AI Agents

One prompt edit quietly turned this customer-support agent into a refund machine. Spooled caught it on the PR.

A PM asks for "a more helpful tone for frustrated customers." An engineer adds one sentence to the system prompt: "Resolve their issue when possible." Unit tests pass. The reviewer approves. The PR is ready to merge.

But the LLM now interprets "resolve" liberally. On complaint tickets, the agent stops escalating refund requests to humans and starts issuing refunds itself. The structure changed even though the prompt looked harmless.

Spooled diffs the agent's behavior against the committed baseline and posts this on the PR:

🚨 Merge blocked: agent now calls `issue_refund`

This tool was never observed in the baseline. It appears in
2 of 5 traces in this PR (~40%).

Triggered by a one-sentence change to the system prompt.

Caught content-blind — Spooled compared tool graphs, not language. It never saw a customer message or an LLM response.

Run it yourself in 60 seconds

pip install spooled-ai
spooled demo

Runs the entire scenario in your terminal — no API key, no setup, no files left behind. The variant agent differs from the baseline by exactly one line in the system prompt. The code is otherwise identical.

What It Does

Capture — wraps your LLM client and records the structural fingerprint of every agent run: which tools were called, in what order, how many times. Content-blind by architecture — prompts, customer data, and AI responses never leave your infrastructure.

Compare — diffs the current run against a committed baseline. Reports structural changes such as tools added or removed, declared tool-schema changes, token shifts, and—when an order-sensitive fingerprint or sequence policy is configured—ordering violations.

Gate — posts a PR comment with the human-readable consequence as the headline. Blocks unprotected traces and high-confidence regressions by default; versioned policy rules define additional application-specific blockers. Resolution instructions are included.

Install

pip install spooled-ai

Quick Start

import spooled
from spooled.wrappers import wrap_openai
from openai import OpenAI

spooled.init(agent_id="my_agent")
client = wrap_openai(OpenAI())

# Calls made through this wrapped client are captured without changing agent logic.
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Analyze this deal"}],
)

spooled.shutdown()

That's it. Exercised calls on the wrapper are captured; instrument tool execution with a supported callback integration or Spooled's tool decorator. The trace is saved to .spooled/traces/. A hash chain links every interaction at capture time; optional attestations provide cryptographic signatures.

CI Integration

Generate the canonical Action workflow for your existing repository:

spooled init project --ci-test-command "python ci_runner.py"

The generated workflow auto-detects requirements.txt, preserves existing workflow files unless --force is supplied, and starts in warn-only shadow mode. After reviewing stable controls and planted regressions, change its blocking input to "true".

Once the agent path is instrumented, add --bootstrap-baseline to run that command three times for training and twice against a disjoint frozen holdout by default. Every admitted behavior must appear in all three training executions, and every holdout trace must match exactly before the first baseline or workflow is retained. The explicit bootstrap invocation approves an immutable governed baseline that records its source revision, disjoint run IDs, and content-blind training/holdout manifests. Bootstrap refuses existing baselines and starts shadow-only; use the governed candidate/approval workflow for later updates. Generated Action jobs have a ten-minute timeout. Use --bootstrap-approved-by when the baseline should record a release-owner identity instead of the repository's configured git user.

# .github/workflows/spooled.yml
- name: Generate traces
  run: python ci_runner.py
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

- name: Spooled behavioral check
  run: |
    pip install spooled-ai
    spooled ci compare .spooled/traces/*.jsonl \
      --baseline .github/baselines \
      --policy spooled-policy.yml \
      --enable-blocking

Example PR comment:

## ❌ Spooled Behavioral CI: FAIL
> Spooled Score: 59/100 (D) 🔴

> [!CAUTION]
> ## 🚨 Merge blocked: agent now calls `issue_refund`
>
> This tool was **never observed in the baseline**. It appears in
> **2 of 5** traces in this PR (~40%).

**5** traces analyzed  |  ✅ **3** passed  |  ❌ **2** policy failures

### Trace Results
| Agent          | Fingerprint     | Status        | Score |
|----------------|-----------------|---------------|-------|
| support_agent  | `4d893b5cef...` | ⚠️ Behavior change | 59 |

<details>
  <summary>🔧 Tool Changes (2 traces)</summary>

  - ➕ `issue_refund` added
  - ➖ `escalate_to_human` removed
</details>

What Spooled Catches

Change type	Example	Unit tests	Spooled
Prompt tweak	"Be concise" drops compliance tools	✅ Pass	Behavior change
Model swap	Model drops sanctions screening	✅ Pass	Behavior change
Tool deprecation	Agent proceeds without critical data	✅ Pass	Behavior change
KB refresh	Ticket response path changes	✅ Pass	Behavior change
Schema migration	Field rename breaks detection	✅ Pass	Behavior change
Upstream degradation	Retry paths appear in fingerprint	✅ Pass	Behavior change

Content-Blind Architecture

Spooled never captures prompts, customer data, or AI responses. Only structural metadata: tool names, call sequence, token counts, timing, plus installation metadata (Spooled version, OS, active hook names, detected framework module names). This is enforced in code — content is stripped before the trace reaches disk. See docs/threat-model.md.

Integration Evidence

Release-proven integration paths:

• OpenAI explicit wrapper
• Anthropic explicit wrapper

These wrappers pass Spooled's complete applicable capture contract against pinned real SDK revisions: sync, async, streaming, cancellation, parallel tool requests, provider errors, application retries, consecutive runs, and structured output. The retained structural receipts were produced with runtime networking disabled and model spend at $0.

Pinned real-runtime workflow proof:

• OpenAI Agents SDK
• LangGraph
• smolagents

These framework adapters pass the seven-case behavioral control/adversary matrix plus their complete applicable capture-family contract against pinned real runtimes. The proof includes consecutive runs, errors and recovery, parallel tools, cancellation, structured output, and real delegation or nested graphs. OpenAI Agents and LangGraph also prove native async and streaming; smolagents proves its native synchronous streaming generator because the pinned agent API has no native async run method. OpenAI Agents cancellation uses Spooled's explicit cancellation helper because the upstream trace object does not expose streaming-result cancellation to processors.

Compatibility-tested, not yet release-proven:

• OpenAI, Anthropic, AWS Bedrock, requests, httpx, and aiohttp global hooks
• LangChain, LlamaIndex, AutoGen, and CrewAI callback integrations

Compatibility-tested surfaces have automated local coverage but have not yet passed the complete pinned-runtime contract. They should not be treated as equivalent to the five release-proven integrations above. See docs/CAPTURE_CONFORMANCE.md and the machine-readable docs/INTEGRATION_SUPPORT.json.

Documentation

• Quick Start
• CI/CD Integration
• Privacy Architecture

License

Proprietary.