AgentDiff

Catch silent cost surges and broken agent loops before they ship.

pip install agent-trajectory-diff

Website & Interactive Docs · Cookbooks · Live Demo Repo · Changelog

AgentDiff is a developer-first Python library, pytest plugin, and CLI for regression testing multi-turn, tool-using AI agents by comparing execution paths (trajectories) head-to-head.

When you change a prompt, tweak a system instruction, or upgrade an LLM, traditional assertions only verify that the final string matches. They miss the silent failures: the agent took 5 extra tool calls, burned 3× the tokens, entered an infinite retry loop, or drifted from the verified execution path.

AgentDiff aligns candidate execution DAGs against committed golden baselines in <10ms without calling any paid LLM judges.

Highlights

• Deterministic Graph Diffing: Topological DAG alignment and Longest Common Subsequence (LCS) step comparison in <10ms with zero paid LLM-judge calls.
• Statistical Baselines & Variance Bands: Capture N-run envelopes (record --runs 3) so non-deterministic agents don't flake CI on harmless jitter.
• Zero-Config Setup (agentdiff init): Auto-detects LangGraph, CrewAI, OpenAI Agents SDK, or OpenTelemetry and writes agentdiff.toml + CI workflow in seconds.
• In-PR Interactive Blessings (/agentdiff approve): Reviewers bless intended trajectory improvements from PR comments as agentdiff-ci[bot].
• 100% Local & Air-Gapped: Zero telemetry, no cloud accounts, no outbound network calls during diffs. Raw prompts and tool outputs stay local.
• Drop-in CI Merge Gate: Native exit codes (0 pass / 1 regression fail) and automated GitHub Action PR comments with human-first verdicts.
• Universal Telemetry Adapters: Seamlessly diff traces from LangGraph, CrewAI, OpenAI Agents SDK, Langfuse, LangSmith, OpenInference / OpenTelemetry, or generic JSON.

Quickstart

1. Initialize with agentdiff init

Auto-detect your agent framework and generate your configuration + CI workflow:

agentdiff init --scenario customer_support --runs 3 --with-approve

2. Record a Statistical Baseline Envelope

Record an N-run baseline envelope from any agent function without writing boilerplate telemetry:

agentdiff record my_agent:run \
  --input '{"query": "summarize repo"}' \
  --runs 3 \
  --out baselines/customer_support.envelope.json

3. Compare Traces in CLI

Compare candidate runs against your baseline envelope:

agentdiff diff baselines/customer_support.envelope.json traces/candidate.json --fail-on-regression

4. Pytest Regression Testing

Enforce trajectory parity directly in your test suite:

import pytest
from agentdiff import load_trace, compare
from agentdiff.testing import assert_no_regressions

def test_agent_refactor_efficiency():
    # Load traces (auto-detects telemetry source format)
    baseline = load_trace("tests/baselines/golden.json")
    candidate = load_trace("tests/traces/candidate.json")

    # Run sub-10ms deterministic comparison
    report = compare(baseline, candidate)

    # Assert no structural drift, cost surges, or tool loops
    assert_no_regressions(
        report,
        max_divergence=0.25,        # Max Trajectory Divergence Index [0.0 - 1.0]
        max_cost_increase_pct=5.0,  # Max 5% token cost increase
        allow_loops=False,          # Reject repetitive tool call cycles
        max_wasted_effort=0.10,     # Max 10% error/retry/abandoned steps
        max_recovery_step_ratio=1.5 # Max recovery steps relative to baseline
    )

GitHub Actions CI Gate

Block broken agent PRs before they land in production using the official composite action:

name: AgentDiff Regression Gate

on:
  pull_request:

permissions:
  contents: read
  pull-requests: write   # Allows posting automated root-cause PR comments

jobs:
  agent-regression-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"

      - uses: kerrshift/agentdiff/.github/actions/agentdiff-check@v0.5.0
        with:
          baseline: baselines/customer_support.envelope.json
          candidate: traces/pr_candidate.json
          pr: ${{ github.event.pull_request.number }}
          github-token: ${{ secrets.GITHUB_TOKEN }}

When a regression occurs, the gate fails with exit code 1 and comments on the PR with culprit identification and a collapsed divergence tree:

### AgentDiff Gate: REGRESSION DETECTED

| Metric | Baseline | Candidate | Threshold | Status |
| :--- | :--- | :--- | :--- | :--- |
| **Divergence (TDI)** | 0.00 | 0.42 | ≤ 0.25 | FAIL |
| **Cost Surge** | $0.0042 | $0.0138 (+228%) | ≤ +5.0% | FAIL |
| **Loops (LBI)** | 0 | 3 loops | 0 | FAIL |
| **Wasted Effort (WEI)**| 0.00 | 0.38 | ≤ 0.10 | FAIL |

**Culprit Step:** Step 4 `execute_sql` entered a 3× retry loop after schema refactor.

Core Metric Mathematics

Supported Telemetry Formats

Local-First Privacy Guarantee

Agent trajectories often contain proprietary prompts, sensitive tool payloads, and customer data. AgentDiff is engineered with strict local-first principles:

• Zero Outbound Network Traffic: Parsing, DAG diffing, metric calculations, and reporting run 100% locally.
• Air-Gapped & Firewall Friendly: Run tests on laptops, in air-gapped VPCs, or under strict enterprise egress policies.
• Repo-Committed Baselines: Your golden trajectories live in Git next to the code they protect.
• No Third-Party APM Lock-In: Switch tracing providers at any time; AgentDiff normalizes all schemas to a unified specification.

Documentation & Cookbooks

• Official Documentation: https://agentdiff.app
• Live Demo Repository: github.com/lostmartian/agentdiff-demo
• Engine Specification: Under the Hood
• Interactive Visualizer: Diff Playground
• Adapters Guide: Framework Integration

Development

This repository uses uv for lightning-fast environment and dependency management.

# Clone the repository
git clone https://github.com/kerrshift/agentdiff.git
cd agentdiff

# Install dependencies and sync virtualenv
uv sync

# Run linting and code formatting checks
make lint

# Run the test suite
make test

# Build package distributions
make build

# Website & docs (separate repo, deploys agentdiff.app)
# → github.com/kerrshift/agentdiff-website

License

Distributed under the MIT License. See LICENSE for more information.