Skip to main content

Agent Diff Python SDK

Python SDK for testing AI agents against isolated replicas of production services.

Installation

pip install agent-diff
# or
uv add agent-diff

Configuration

Option 1: Environment Variables

Set these environment variables and the SDK will use them automatically:

export AGENT_DIFF_API_KEY="ad_live_sk_..."
export AGENT_DIFF_BASE_URL="https://api.agentdiff.dev/api/platform"

Then initialize the client without arguments:

from agent_diff import AgentDiff

client = AgentDiff()  # Reads from environment variables

Local Development

For self-hosted instances, point to your local server:

client = AgentDiff(base_url="http://localhost:8000")

Environments

Create isolated, ephemeral replicas of services:

env = client.init_env(
    templateService="slack",
    templateName="slack_default",
    impersonateUserId="U123",
    ttlSeconds=3600
)

# Access environment details
env.environmentId
env.environmentUrl
env.expiresAt

# Delete when done
client.delete_env(env.environmentId)

Templates

List and create environment templates:

# List available templates
templates = client.list_templates()

# Create custom template - you can populate the replica via API and turn it into a template with custom data
custom = client.create_template_from_environment(
    environmentId=env.environmentId,
    service="slack",
    name="my_template",
    description="Custom template",
    visibility="private"  # "private" means only you can view the template
)

Code Execution Proxies

SDK provides code execution proxies that automatically intercept API calls and route them to isolated test environments. This enables agents with code execution capabilities to interact with service replicas without any code changes.

How It Works

When your agent executes Python or Bash code:

  1. The executor wraps your code with interception logic
  2. API calls to https://slack.com/api/* → routed to your sandbox
  3. API calls to https://api.linear.app/* → routed to your sandbox
  4. API calls to https://api.box.com/2.0/* → routed to your sandbox
  5. Your agent sees real API responses from the isolated environment

Important: Executor Configuration

Executors run code in a subprocess, so environment variables from your main process don't automatically transfer. Always pass base_url and api_key explicitly:

executor = PythonExecutorProxy(
    env.environmentId,
    base_url=client.base_url,
    api_key=client.api_key
)

executor = PythonExecutorProxy(env.environmentId)

Available Executors

PythonExecutorProxy

Intercepts Python requests and urllib libraries:

from agent_diff import PythonExecutorProxy, create_openai_tool

python_executor = PythonExecutorProxy(
    env.environmentId,
    base_url=client.base_url,
    api_key=client.api_key
)
python_tool = create_openai_tool(python_executor)

# Works with OpenAI Agents SDK, LangChain, smolagents
agent = Agent(
    model="gpt-4o",
    tools=[python_tool],
    instructions="Use execute_python tool to interact with Slack API at https://slack.com/api/*. Authentication is automatic."
)
agent.run("Send a Slack message to #general")

BashExecutorProxy

Intercepts curl commands:

from agent_diff import BashExecutorProxy, create_openai_tool

bash_executor = BashExecutorProxy(
    env.environmentId,
    base_url=client.base_url,
    api_key=client.api_key
)
bash_tool = create_openai_tool(bash_executor)

agent = Agent(
    model="gpt-4o",
    tools=[bash_tool],
    instructions="Use execute_bash tool with curl to interact with Slack API at https://slack.com/api/*. Authentication is automatic."
)
agent.run("Use curl to post a message to Slack")

Framework Support

Create tools for popular agent frameworks:

from agent_diff import create_openai_tool, create_langchain_tool, create_smolagents_tool

# OpenAI Agents SDK
openai_tool = create_openai_tool(python_executor)

# LangChain
langchain_tool = create_langchain_tool(python_executor)

# HuggingFace smolagents
smolagents_tool = create_smolagents_tool(python_executor)

Direct Execution

For custom frameworks or direct usage:

python_executor = PythonExecutorProxy(
    env.environmentId,
    base_url=client.base_url,
    api_key=client.api_key
)

result = python_executor.execute("""
import requests
response = requests.post('https://slack.com/api/chat.postMessage', json={
    'channel': '#general',
    'text': 'Hello from Agent Diff!'
})
print(response.json())
""")

if result["status"] == "success":
    print(result["stdout"])
else:
    print(result["stderr"])

Test Suites & Evaluations

To run evaluations:

suite_list = client.list_test_suites(name="Slack Bench")
slack_suite = suite_list.testSuites[0]
test_suite = client.get_test_suite(slack_suite.id, expand=True)

evaluation_results = []


for test in test_suite.tests:
    prompt = test.prompt
    test_id = test.id

    env = client.init_env(testId=test_id)
    run = client.start_run(envId=env.environmentId, testId=test_id)

    # Create executor with automatic API interception
    python_executor = PythonExecutorProxy(env.environmentId, base_url=client.base_url)
    python_tool = create_openai_tool(python_executor)

    # Run your agent with the tool
    agent = Agent(
        model="gpt-4o",
        tools=[python_tool],
        instructions="Use execute_python to interact with Slack at https://slack.com/api/*. Authentication is automatic."
    )
    response = agent.run(prompt)

    evaluation_result = client.evaluate_run(runId=run.runId)  # Returns score, runId, status and Score (0/1)

    evaluation_results.append(evaluation_result)

    client.delete_env(envId=env.environmentId)

License

MIT License - see LICENSE file for details.

Metadata

Release files for agent-diff 1.0.7

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for agent-diff 1.0.7
File Size Uploaded Provenance Actions
agent_diff-1.0.7.tar.gz 19.8 kB Without provenance Details for agent_diff-1.0.7.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for agent-diff 1.0.7
File Interpreter ABI Platform Actions
agent_diff-1.0.7-py3-none-any.whl Python 3 none any Details for agent_diff-1.0.7-py3-none-any.whl

Total release size: 38.7 kB

Release files / agent_diff-1.0.7.tar.gz

Download URL agent_diff-1.0.7.tar.gz
Size 19.8 kB
Tags Source
SHA-256 checksum
How to use checksums
4de0b07acb7267c565edb3a3b1804ca97b8d1101fc1431067894c3d14fad2c29
BLAKE2b-256 checksum
How to use checksums
95e4cd2132f4bdb3b5f1df05c6f1ee43097de610e22801de224269fa5729e896
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.9.2

Release files / agent_diff-1.0.7-py3-none-any.whl

Download URL agent_diff-1.0.7-py3-none-any.whl
Size 18.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
37a5db0ac7c5460f2e92e6f0b75d456fadcb2391ee11f04e110d32963581b0d0
BLAKE2b-256 checksum
How to use checksums
3062e94b994aff74d9b57b21f1bfc0baaba2184a41ccf3adebd91cc983f6fa45
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.9.2

Security for agent-diff 1.0.7

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.0.7 This release

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page