Agent Diff Python SDK
Python SDK for testing AI agents against isolated replicas of production services.
Installation
pip install agent-diff
# or
uv add agent-diff
Configuration
Option 1: Environment Variables
Set these environment variables and the SDK will use them automatically:
export AGENT_DIFF_API_KEY="ad_live_sk_..."
export AGENT_DIFF_BASE_URL="https://api.agentdiff.dev/api/platform"
Then initialize the client without arguments:
from agent_diff import AgentDiff
client = AgentDiff() # Reads from environment variables
Local Development
For self-hosted instances, point to your local server:
client = AgentDiff(base_url="http://localhost:8000")
Environments
Create isolated, ephemeral replicas of services:
env = client.init_env(
templateService="slack",
templateName="slack_default",
impersonateUserId="U123",
ttlSeconds=3600
)
# Access environment details
env.environmentId
env.environmentUrl
env.expiresAt
# Delete when done
client.delete_env(env.environmentId)
Templates
List and create environment templates:
# List available templates
templates = client.list_templates()
# Create custom template - you can populate the replica via API and turn it into a template with custom data
custom = client.create_template_from_environment(
environmentId=env.environmentId,
service="slack",
name="my_template",
description="Custom template",
visibility="private" # "private" means only you can view the template
)
Code Execution Proxies
SDK provides code execution proxies that automatically intercept API calls and route them to isolated test environments. This enables agents with code execution capabilities to interact with service replicas without any code changes.
How It Works
When your agent executes Python or Bash code:
- The executor wraps your code with interception logic
- API calls to
https://slack.com/api/*→ routed to your sandbox - API calls to
https://api.linear.app/*→ routed to your sandbox - API calls to
https://api.box.com/2.0/*→ routed to your sandbox - Your agent sees real API responses from the isolated environment
Important: Executor Configuration
Executors run code in a subprocess, so environment variables from your main process don't automatically transfer. Always pass base_url and api_key explicitly:
executor = PythonExecutorProxy(
env.environmentId,
base_url=client.base_url,
api_key=client.api_key
)
executor = PythonExecutorProxy(env.environmentId)
Available Executors
PythonExecutorProxy
Intercepts Python requests and urllib libraries:
from agent_diff import PythonExecutorProxy, create_openai_tool
python_executor = PythonExecutorProxy(
env.environmentId,
base_url=client.base_url,
api_key=client.api_key
)
python_tool = create_openai_tool(python_executor)
# Works with OpenAI Agents SDK, LangChain, smolagents
agent = Agent(
model="gpt-4o",
tools=[python_tool],
instructions="Use execute_python tool to interact with Slack API at https://slack.com/api/*. Authentication is automatic."
)
agent.run("Send a Slack message to #general")
BashExecutorProxy
Intercepts curl commands:
from agent_diff import BashExecutorProxy, create_openai_tool
bash_executor = BashExecutorProxy(
env.environmentId,
base_url=client.base_url,
api_key=client.api_key
)
bash_tool = create_openai_tool(bash_executor)
agent = Agent(
model="gpt-4o",
tools=[bash_tool],
instructions="Use execute_bash tool with curl to interact with Slack API at https://slack.com/api/*. Authentication is automatic."
)
agent.run("Use curl to post a message to Slack")
Framework Support
Create tools for popular agent frameworks:
from agent_diff import create_openai_tool, create_langchain_tool, create_smolagents_tool
# OpenAI Agents SDK
openai_tool = create_openai_tool(python_executor)
# LangChain
langchain_tool = create_langchain_tool(python_executor)
# HuggingFace smolagents
smolagents_tool = create_smolagents_tool(python_executor)
Direct Execution
For custom frameworks or direct usage:
python_executor = PythonExecutorProxy(
env.environmentId,
base_url=client.base_url,
api_key=client.api_key
)
result = python_executor.execute("""
import requests
response = requests.post('https://slack.com/api/chat.postMessage', json={
'channel': '#general',
'text': 'Hello from Agent Diff!'
})
print(response.json())
""")
if result["status"] == "success":
print(result["stdout"])
else:
print(result["stderr"])
Test Suites & Evaluations
To run evaluations:
suite_list = client.list_test_suites(name="Slack Bench")
slack_suite = suite_list.testSuites[0]
test_suite = client.get_test_suite(slack_suite.id, expand=True)
evaluation_results = []
for test in test_suite.tests:
prompt = test.prompt
test_id = test.id
env = client.init_env(testId=test_id)
run = client.start_run(envId=env.environmentId, testId=test_id)
# Create executor with automatic API interception
python_executor = PythonExecutorProxy(env.environmentId, base_url=client.base_url)
python_tool = create_openai_tool(python_executor)
# Run your agent with the tool
agent = Agent(
model="gpt-4o",
tools=[python_tool],
instructions="Use execute_python to interact with Slack at https://slack.com/api/*. Authentication is automatic."
)
response = agent.run(prompt)
evaluation_result = client.evaluate_run(runId=run.runId) # Returns score, runId, status and Score (0/1)
evaluation_results.append(evaluation_result)
client.delete_env(envId=env.environmentId)
License
MIT License - see LICENSE file for details.
Metadata
Release files for agent-diff 1.0.7
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| agent_diff-1.0.7.tar.gz | 19.8 kB | Without provenance | Details for agent_diff-1.0.7.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| agent_diff-1.0.7-py3-none-any.whl | Python 3 | none | any | Details for agent_diff-1.0.7-py3-none-any.whl |
Total release size: 38.7 kB
Release files / agent_diff-1.0.7.tar.gz
| Download URL | agent_diff-1.0.7.tar.gz |
|---|---|
| Size | 19.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
4de0b07acb7267c565edb3a3b1804ca97b8d1101fc1431067894c3d14fad2c29
|
|
BLAKE2b-256 checksum How to use checksums |
95e4cd2132f4bdb3b5f1df05c6f1ee43097de610e22801de224269fa5729e896
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.9.2
|
Release files / agent_diff-1.0.7-py3-none-any.whl
| Download URL | agent_diff-1.0.7-py3-none-any.whl |
|---|---|
| Size | 18.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
37a5db0ac7c5460f2e92e6f0b75d456fadcb2391ee11f04e110d32963581b0d0
|
|
BLAKE2b-256 checksum How to use checksums |
3062e94b994aff74d9b57b21f1bfc0baaba2184a41ccf3adebd91cc983f6fa45
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.9.2
|
Security for agent-diff 1.0.7
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.