spooled-ai 0.4.4

CI for AI agents - behavioral fingerprinting and drift detection

Spooled — Behavioral CI for AI Agents

The diff for agent behavior. Capture what your agent does, detect when it changes, gate the PR.

AI agents are non-deterministic. The same code, prompt, and model produce different tool-calling behavior on every run. A one-word prompt edit can silently drop a compliance check. A model upgrade can change which tools get called. A KB refresh can alter the agent's decision path. Unit tests pass. Eval suites pass. Nobody notices until production.

Spooled catches it on the PR.

What It Does

Capture — wraps your LLM client and records the structural fingerprint of every agent run: which tools were called, in what order, how many times. Content-blind by architecture — prompts, customer data, and AI responses never leave your infrastructure.

Compare — diffs the current run against a committed baseline. Shows exactly what changed: tools added, tools removed, sequence reordered, token usage shifted.

Gate — posts a PR comment with a behavioral score. Blocks the merge if the policy says so. Resolution instructions included.

Install

pip install spooled-ai

Quick Start

import spooled
from spooled.wrappers import wrap_openai
from openai import OpenAI

spooled.init(agent_id="my_agent")
client = wrap_openai(OpenAI())

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Analyze this deal"}],
    tools=MY_TOOLS,
)

spooled.shutdown()

That's it. Every tool call is captured. The trace is saved to .spooled/traces/. The hash chain signs every interaction at capture time.

CI Integration

# .github/workflows/spooled.yml
- name: Generate traces
  run: python ci_runner.py
  env:
    OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}

- name: Spooled behavioral check
  run: |
    pip install spooled-ai
    spooled ci compare .spooled/traces/*.jsonl \
      --baseline .github/baselines \
      --policy spooled-policy.yml \
      --enable-blocking

Example PR comment:

❌ Spooled Behavioral CI: FAIL
Spooled Score: 59/100 (D) 🔴

| Agent      | Status                      | Score | Tokens            |
|------------|-----------------------------|-------|-------------------|
| deal_agent | ⚠️ Variant · Tooling change | 59    | 🔻 14198 (-32%)   |

🔧 Tool Changes:
  ➖ sanctions_screening removed
  ➖ ip_patent_search removed

What Spooled Catches

Change type	Example	Unit tests	Spooled
Prompt tweak	"Be concise" drops compliance tools	✅ Pass	VARIANT
Model swap	Model drops sanctions screening	✅ Pass	VARIANT
Tool deprecation	Agent proceeds without critical data	✅ Pass	VARIANT
KB refresh	Ticket response path changes	✅ Pass	VARIANT
Schema migration	Field rename breaks detection	✅ Pass	VARIANT
Upstream degradation	Retry paths appear in fingerprint	✅ Pass	VARIANT

Content-Blind Architecture

Spooled never captures prompts, customer data, or AI responses. Only structural metadata: tool names, call sequence, token counts, timing. This is enforced in code — content is stripped before the trace reaches disk.

Supported Libraries

LLM Providers (explicit wrappers):

• OpenAI (sync/async, streaming)
• Anthropic (sync/async, streaming)

HTTP & Cloud (auto-instrumented via hooks):

• AWS Bedrock
• requests, httpx, aiohttp

Frameworks (callback handlers):

• LangChain, LlamaIndex, AutoGen, CrewAI, LangGraph

Documentation

• Quick Start
• CI/CD Integration
• Privacy Architecture

License

Proprietary.