🛡️ Fixtura

Deterministic Execution Recording, Replay, & Drift Testing for AI Agents — Turn Real Agent Runs into Replayable Test Fixtures.

---

🚀 Welcome to Fixtura! Fixtura gives your AI agents record & replay superpowers. Capture real agent behavior once, redact sensitive secrets automatically, and replay those traces deterministically in CI/CD — with zero live API costs and zero flaky external calls!

---

⚠️ Upgrading from 1.x? Imports moved.

2.0.0 puts everything under the fixtura. namespace. Versions before 2.0 published recorder, replay, security, tools, cli, and analysis as top-level packages, so installing fixtura claimed six generic names on your sys.path and could shadow your own tools/ or cli/ package.

Prefix your imports with fixtura.:

- from recorder.recorder import ExecutionRecorder
+ from fixtura.recorder.recorder import ExecutionRecorder

Also update module paths written as strings, such as mock.patch("tools.base_tool.check"). The CLI, its subcommands, and the .trace format are unchanged. Full table in CHANGELOG.md.

Pinning fixtura>=1.x with no upper bound will resolve 2.0.0 and break — use fixtura>=1.1.1,<2 if you are not ready to migrate.

---

✨ Features At A Glance — Why You'll Love Fixtura

• ⚡ Lightning Fast Offline Replays: Reproduce complex multi-step agent runs in milliseconds without calling live LLM models or touching real databases.
• 🔒 Ironclad Security & Gatekeeping: Capability tokens enforce granular read/write permissions at the tool execution boundary before side effects ever execute.
• 🧼 Automatic Secret Redaction: Built-in sanitizer scrubs planted API keys, passwords, and tokens before .trace files are stored.
• 🚨 Automated Fixture Drift Shield: Structural fingerprinting detects if a prompt or tool schema changed, preventing stale fixtures from giving false greens in CI.
• 📊 Boundary Coverage Radar: Instantly visualize tool coverage and boundary safety (permission denials, rate limits, errors).
• 🔌 100% Framework Agnostic: Plugs seamlessly underneath LangGraph, CrewAI, AutoGen, or custom agent loops in under 5 lines of code.

---

💡 What is Fixtura?

Fixtura is a framework-agnostic execution-assurance layer for AI agents. Sitting directly at the tool-execution boundary, Fixtura monitors an agent's tool calls, enforces real-time permission policies and rate limits, redacts secrets, records deterministic .trace files, and allows you to replay, diff, and grade those recordings offline without touching live systems.

       ┌────────────────┐
       │   AI  Agent    │ (LangGraph / CrewAI / AutoGen / Custom)
       └───────┬────────┘
               │ executor.call("tool_name", args)
               ▼
┌───────────────────────────────┐
│     Fixtura Executor Seam     │
├──────────────┬────────────────┤
│ Permission   │ Rate Limiter   │
│ Engine       │ & Circuit Bkr  │
└──────┬───────┴────────┬───────┘
       │                │
       ▼                ▼
┌──────────────┐ ┌──────────────┐
│  Real Tools  │ │ Execution    │ ───►  saved  .trace  artifact
│ (FS/SQL/HTTP)│ │ Recorder     │       (compressed & redacted)
└──────────────┘ └──────────────┘

---

🎯 Why Fixtura vs Production APM?

Most agent observability tools (Langfuse, Phoenix, Braintrust) focus on production monitoring and LLM latency metrics at scale. Fixtura takes a different approach: recorded execution traces as literal, replayable test fixtures.

Metric	Production APM (Langfuse/Phoenix)	🛡️ Fixtura Execution Assurance
Primary Goal	Telemetry, cost tracking, prompt latency	CI/CD regression testing & deterministic verification
Execution Control	Passive monitoring (read-only telemetry)	Active permission gatekeeper & circuit breaker
Replay Mechanism	Visual timeline inspection	Offline Passive & Verified Replay (zero live calls)
Regression Testing	LLM-as-a-judge scoring	Structural fingerprint drift detection (check-drift)
CI Integration	Post-hoc dashboards	Binary exit codes (0 PASS, 1 DRIFTED, 2 UNVERIFIED)

---

🏗️ System Architecture & Dataflow

flowchart TD
    classDef agent fill:#e1f5fe,stroke:#0288d1,stroke-width:2px,color:#01579b;
    classDef seam fill:#fff3e0,stroke:#f57c00,stroke-width:2px,color:#e65100;
    classDef gate fill:#ffebee,stroke:#c62828,stroke-width:2px,color:#b71c1c;
    classDef storage fill:#ede7f6,stroke:#512da8,stroke-width:2px,color:#311b92;
    classDef engine fill:#e8f5e9,stroke:#388e3c,stroke-width:2px,color:#1b5e20;

    subgraph "Agent Runtime Layer"
        LLM["🤖 Agent Model / Orchestrator"]:::agent
    end

    subgraph "Fixtura Execution Seam"
        Exec["⚡ FixturaExecutor (Injected Seam)"]:::seam
    end

    subgraph "Protection & Security Gatekeeper"
        Perm["🔒 Permission Engine (Capability Token)"]:::gate
        Limiter["⏱️ Rate Limiter & Circuit Breaker"]:::gate
        Sanitizer["🧼 Secret Sanitizer (Redactor)"]:::gate
    end

    subgraph "Trace Artifacts"
        Rec["REC Execution Recorder"]:::storage
        TraceFile[("💾 .trace Artifact (Zstd Compressed)")]:::storage
    end

    subgraph "Verification & Replay Suite"
        Replay["⏪ Passive Replay"]:::engine
        Inspect["🔍 Step Inspector / UI"]:::engine
        Drift["🚨 check-drift (Fingerprint Verifier)"]:::engine
        Verify["✅ Verified Replay (Diff Engine)"]:::engine
        Coverage["📊 Fixture Library Coverage"]:::engine
        OpenEval["💯 OpenEval Adapter"]:::engine
    end

    LLM -->|"1. executor.call(tool, args)"| Exec
    Exec -->|"2. Authorize"| Perm
    Exec -->|"3. Quota check"| Limiter
    Perm -->|"4. Sanitize input/output"| Sanitizer
    Sanitizer -->|"5. Write event"| Rec
    Rec -->|"6. Produce"| TraceFile

    TraceFile --> Replay
    TraceFile --> Inspect
    TraceFile --> Drift
    TraceFile --> Verify
    TraceFile --> Coverage
    TraceFile --> OpenEval

---

🚀 Quickstart (5 Minutes)

1. Installation

# Install from PyPI:
pip install fixtura

# Or install in editable mode from a local checkout:
git clone https://github.com/yash161004/fixtura-core.git
cd fixtura-core
pip install -e .

2. Run the Interactive Quickstart Script

python quickstart.py

The script automatically demonstrates the complete 6-stage pipeline:

1. 🎬 Record: Captures a multi-step agent run to quickstart_example.trace.
2. ⏪ Passive Replay: Reproduces the run offline with zero live calls.
3. 🔍 Step Inspection: Interactively steps through each recorded event.
4. 🚨 Drift Check: Verifies trace structural fingerprints against current tool schemas and policies.
5. ✅ Verified Replay: Re-executes agent logic while substituting tool call responses.
6. 📊 Coverage Analysis: Computes tool and boundary coverage across the fixture library.

---

🔌 Wiring Fixtura into Your Agent

Fixtura integrates into your existing agent loop with zero structural refactoring. Simply inject fixtura.executor where your agent executes tools:

import fixtura

# 1. Initialize executor in record or replay mode
executor = fixtura.executor(
    mode="record",  # Or mode="replay" for offline runs
    trace="agent_run.trace",
    tools=tool_registry,
    capability_token=capability_token,
)

# 2. Call tools inline from inside your agent loop
result = executor.call("filesystem_tool", {"operation": "read", "path": "notes.txt"})

if result.denied:
    print(f"🔒 Permission denied: {result.reason}")
else:
    print(f"✅ Tool output: {result.value}")

Execution Modes:

• Record Mode (mode="record"): Executes tools for real through the permission engine and rate limiter, sanitizing outputs and writing every event to .trace.
• Replay Mode (mode="replay"): Returns recorded outputs inline with zero live tool execution. No capability token or real tools needed.

---

⚡ Core Features & Capabilities

1. 🚨 Fixture Drift Detection (check-drift)

A test fixture is only useful if it matches current code. If you update a tool's Pydantic schema or edit a system prompt, a stale trace replaying green creates a false sense of security.

Fixtura traces carry a cryptographic fingerprint of tool schemas, prompt text, and policy rules. check-drift detects changes before replaying:

fixtura check-drift fixtures/checkout.trace --agent my_agent:spec

DRIFTED: fixtures/checkout.trace
  agent:   my_agent:spec
  changed: tool schema changed: sqlite_tool

• Exit Codes: 0 PASS, 1 DRIFTED, 2 UNVERIFIED (malformed/unimportable spec).
• Note: Run from your project root or set PYTHONPATH=. so Python can resolve your custom AgentSpec.

2. ✅ Verified Replay (verify)

Verified Replay runs your agent's decision logic live while substituting tool call responses from the trace. It diffs generated tool requests against recorded tool requests while normalizing volatile fields (such as timestamps, UUIDs, and transient IDs).

fixtura verify fixtures/checkout.trace

3. 📊 Fixture Library Coverage (coverage)

Analyzes a corpus of trace files to report which registered tools have been tested, and whether boundary conditions (permission denials, errors, rate limits) have been exercised.

fixtura coverage fixtures/

Coverage over 4 trace(s), 12 recorded tool call(s)

  TOOL             CALLS  OUTCOMES SEEN
  ---------------  -----  ----------------------------------------
  filesystem_tool      4  allowed, denied
  sqlite_tool          5  allowed, error
  http_tool            3  denied, throttled

  Tool coverage:     100.0%  (registered tools appearing in any fixture)
  Boundary coverage:  83.3%  (registered tools seen failing, denied, or throttled)

4. 🔀 The 4 Replay Modes

matrix
    title Replay Mode Spectrum
    "Mode" : "Executes Tools?" : "Agent Runs?" : "Deterministic?"
    "Passive Replay" : "No" : "No" : "Yes (100%)"
    "Step Inspector" : "No" : "No" : "Yes (100%)"
    "Verified Replay" : "No" : "Yes" : "Partial (Tool-level)"
    "Live Branching" : "Yes" : "Yes" : "No (Live Run)"

Mode	Command	Agent Runs?	Touches Live Systems?	Compares Output?
Passive Replay	fixtura replay <trace>	No	No	No
Step Inspection	fixtura inspect <trace>	No	No	No
Verified Replay	fixtura verify <trace>	Yes	No	Yes
Live Branching	fixtura branch <parent> <out> <step> <prompt>	Yes	Yes (Live)	No

---

🛠️ CLI Reference Manual

Command	Arguments	Description	Exit Code
fixtura record	<trace>	Record an agent run to a .trace artifact using demo agent	0
fixtura replay	<trace>	Passively replay a trace offline with zero live calls	0
fixtura inspect	<trace>	Step through a trace interactively line-by-line	0
fixtura view	<trace>	Print human-readable summary of recorded events	0
fixtura html-view	<trace>	Generate standalone HTML trace visualizer (trace_viewer.html)	0
fixtura check-drift	<trace> [--agent module:attr]	Verify if trace fingerprint matches current agent spec	0 PASS / 1 DRIFTED / 2 UNVERIFIED
fixtura verify	<trace> [prompt]	Run agent decision logic offline & diff tool calls	0 PASS / 1 MISMATCH / 2 DRIFTED
fixtura coverage	<paths...>	Report tool and boundary coverage across .trace files	0
fixtura branch	<parent> <out> <step> <prompt>	Branch trace execution from a step (executes live)	0
fixtura eval	<trace>	Score trace trajectory via OpenEval adapter	0

---

🤖 Running Real Agent Loops

Fixtura includes a complete, provider-agnostic real agent loop in examples/real_agent.py. It works against any OpenAI-compatible LLM endpoint (Ollama, Groq, OpenRouter, Together, Gemini, OpenAI):

Free Local Execution (Ollama):

ollama pull qwen3:4b
ollama serve
python -m examples.real_agent run.trace

Hosted Provider Execution:

export FIXTURA_LLM_BASE_URL="https://api.groq.com/openai/v1"
export FIXTURA_LLM_MODEL="llama-3.3-70b-versatile"
export FIXTURA_LLM_API_KEY="your-api-key"

python -m examples.real_agent run.trace

---

💯 OpenEval Adapter (Optional)

Fixtura integrates with OpenEval for trajectory scoring. OpenEval is published on PyPI as openeval-core, so the eval extra installs it for you:

# Install Fixtura with the OpenEval adapter:
pip install 'fixtura[eval]'

# Score a trace trajectory:
fixtura eval quickstart_example.trace

Or add OpenEval to an existing install:

pip install 'openeval-core>=0.2.1'

Note on the distribution name. The adapter requires openeval-core, not openeval. The bare openeval name on PyPI is an unrelated placeholder project that installs cleanly and does nothing. The native Fixtura adapter (openeval.adapters.fixtura) ships from openeval-core 0.2.1 onward; earlier releases do not contain it, which is why the extra pins >=0.2.1.

---

📚 Repository Documentation Index

Document	Description
📐 ARCHITECTURE.md	Detailed system design, trust boundaries, and scope decisions
🛡️ THREAT_MODEL.md	Security model, prompt injection defenses, secret sanitization
🗺️ ROADMAP.md	Milestone tracking, frozen scope table, and acceptance criteria
🚦 PROJECT_STATUS.md	Comprehensive test suite matrix and component build status
📋 CHANGELOG.md	Release notes and version history
📄 TRACE_FORMAT_SPEC.md	Specification of the .trace binary format and JSON schema
🔬 LIVE_RUN_FINDINGS.md	Empirical findings and fixes from live LLM model testing
📜 DENIAL_MESSAGE_PROPOSAL.md	Security proposal on permission-denial message granularity

---

🤝 Community & Contributions

We welcome feedback, issues, and contributions from developer teams building reliable AI agents!

• ⭐ Star the repo if Fixtura helps you catch agent regressions in CI!
• 💬 Open an Issue for suggestions, bug reports, or feature ideas.
• 📜 See LICENSE for MIT licensing details.

---

📄 License

This project is licensed under the MIT License — see the LICENSE file for details.