AgentDrill
Test what your AI agent actually changes, not just what it says.
AgentDrill is an open-source Python testing framework for tool-using AI agents.
It runs agents against simulated environments and independently verifies resulting business-state changes.
Instead of asking only whether an AI agent produced a correct response, AgentDrill checks whether its actions respected business rules.
Why AgentDrill?
An AI agent might:
- Transfer the wrong amount of money
- Issue a duplicate refund
- Modify an unauthorized customer field
- Reserve incorrect inventory quantities
- Fail to complete a legitimate operation
AgentDrill helps developers detect these problems during testing, before connecting agents to real business systems.
Features
- Bring your own Python agent
- Define custom simulated environments
- Register and trace Python tool calls
- Inspect state before and after execution
- Define custom business-rule verification
- Detect incorrect and incomplete actions
- Generate structured JSON reports
- Run tests through a Python API or CLI
Installation
Requires Python 3.10 or later.
Clone the repository and install:
git clone https://github.com/agentdrillhq/agentdrill.git
cd agentdrill
python -m pip install -e .
The repository URL will be updated after the GitHub repository is created.
Quick Start
Run the bundled generic example:
agentdrill custom --file examples/quickstart.py
Expected output:
AGENTDRILL - CUSTOM AGENT EVALUATION
Scenario: custom_transfer_test
Agent: Agent
Result: PASS
Failed rules: []
Tool events: 1
Python API
Import the public interfaces:
from agentdrill import Scenario, evaluate, ToolRegistry
Define a scenario:
scenario = Scenario(
name="my_workflow",
request="Perform the authorized action",
setup=MyWorld,
make_tools=make_tools,
verify=my_verifier,
)
Evaluate an agent:
result = evaluate(MyAgent(), scenario)
print(result["passed"])
print(result["verification"])
The developer supplies the world, tools, agent, and verification rules.
Required Interfaces
Environment
The environment implements snapshot():
class MyWorld:
def snapshot(self):
return {"value": 10}
Agent
The agent implements run(tools, request):
class MyAgent:
def run(self, tools, request):
return tools.my_action(value=5)
Verification
The verifier receives before, after, and world:
def my_verifier(before, after, world):
passed = after["value"] == 5
return {
"passed": passed,
"failed_rules": [] if passed else ["wrong_value"],
}
Generic Tool Tracing
Register tools through ToolRegistry:
registry = ToolRegistry(snapshot=world.snapshot)
registry.register(
"my_action",
world.my_action,
)
registry.call("my_action", value=5)
print(registry.events)
The registry records:
- Tool name and arguments
- Execution result or exception
- State before and after execution
- Whether the state changed
- Execution duration
CLI
Run a custom scenario:
agentdrill custom --file my_test.py
The Python file must define:
- scenario: an AgentDrill Scenario instance
- Agent: a Python class implementing run(tools, request)
List bundled refund scenarios:
agentdrill list
Test a compatible refund agent:
agentdrill test --agent examples/my_agent.py
The generic Python API is domain-independent. The bundled refund CLI remains a domain-specific example.
Examples
The repository includes demonstrations for:
- Refund workflows
- CRM customer updates
- Inventory management
- Banking-style transfers
- Custom business rules
Limitations
AgentDrill is currently an experimental alpha.
- Agent code runs in the same Python process.
- It is not a security sandbox.
- Custom simulations and verification rules are developer-defined.
- Tool tracing only covers calls routed through the tracing adapter.
- Arbitrary third-party agent frameworks are not automatically supported.
- Automatic adversarial scenario generation is not yet available.
- Production reliability has not been established.
Do not use the current execution model to run untrusted Python code or connect tests directly to irreversible production actions.
Project Status
Version 0.7.0 — Alpha.
We are looking for developers building tool-using AI agents who want to test business-state correctness.
Feedback and reproducible failure cases are especially valuable.
License
MIT.
Metadata
Release files for agentdrill 0.7.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| agentdrill-0.7.0.tar.gz | 20.7 kB | With provenance | Details for agentdrill-0.7.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| agentdrill-0.7.0-py3-none-any.whl | Python 3 | none | any | Details for agentdrill-0.7.0-py3-none-any.whl |
Total release size: 48.1 kB
Release files / agentdrill-0.7.0.tar.gz
| Download URL | agentdrill-0.7.0.tar.gz |
|---|---|
| Size | 20.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
521ee87af7035e7c1f54e12e01ebfa3376179b8e6a878d42257cd4c3cb4fb8cd
|
|
BLAKE2b-256 checksum How to use checksums |
a789dc066d1f70582f239c86d34ab0ff059ab3e64f130d09166d8140f2c3a919
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 22, 2026.
Transparency logRelease files / agentdrill-0.7.0-py3-none-any.whl
| Download URL | agentdrill-0.7.0-py3-none-any.whl |
|---|---|
| Size | 27.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
56087b9bf34eeccdf7ee75075b1e1c97c21828554c654ace2d69c2fdb8a55b3f
|
|
BLAKE2b-256 checksum How to use checksums |
b213991d6cb5ded350e0f2a98f1aa0c7a6707b920933c42c79df62a43d6fc1cd
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/7.0.0 CPython/3.13.16
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 22, 2026.
Transparency logSecurity for agentdrill 0.7.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for agentdrill-0.7.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Oct 22, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.