Skip to main content

AgentDrill

Test what your AI agent actually changes, not just what it says.

AgentDrill is an open-source Python testing framework for tool-using AI agents.

It runs agents against simulated environments and independently verifies resulting business-state changes.

Instead of asking only whether an AI agent produced a correct response, AgentDrill checks whether its actions respected business rules.

Why AgentDrill?

An AI agent might:

  • Transfer the wrong amount of money
  • Issue a duplicate refund
  • Modify an unauthorized customer field
  • Reserve incorrect inventory quantities
  • Fail to complete a legitimate operation

AgentDrill helps developers detect these problems during testing, before connecting agents to real business systems.

Features

  • Bring your own Python agent
  • Define custom simulated environments
  • Register and trace Python tool calls
  • Inspect state before and after execution
  • Define custom business-rule verification
  • Detect incorrect and incomplete actions
  • Generate structured JSON reports
  • Run tests through a Python API or CLI

Installation

Requires Python 3.10 or later.

Clone the repository and install:

git clone https://github.com/agentdrillhq/agentdrill.git
cd agentdrill
python -m pip install -e .

The repository URL will be updated after the GitHub repository is created.

Quick Start

Run the bundled generic example:

agentdrill custom --file examples/quickstart.py

Expected output:

AGENTDRILL - CUSTOM AGENT EVALUATION

Scenario: custom_transfer_test
Agent: Agent

Result: PASS
Failed rules: []
Tool events: 1

Python API

Import the public interfaces:

from agentdrill import Scenario, evaluate, ToolRegistry

Define a scenario:

scenario = Scenario(
    name="my_workflow",
    request="Perform the authorized action",
    setup=MyWorld,
    make_tools=make_tools,
    verify=my_verifier,
)

Evaluate an agent:

result = evaluate(MyAgent(), scenario)

print(result["passed"])
print(result["verification"])

The developer supplies the world, tools, agent, and verification rules.

Required Interfaces

Environment

The environment implements snapshot():

class MyWorld:
    def snapshot(self):
        return {"value": 10}

Agent

The agent implements run(tools, request):

class MyAgent:
    def run(self, tools, request):
        return tools.my_action(value=5)

Verification

The verifier receives before, after, and world:

def my_verifier(before, after, world):
    passed = after["value"] == 5

    return {
        "passed": passed,
        "failed_rules": [] if passed else ["wrong_value"],
    }

Generic Tool Tracing

Register tools through ToolRegistry:

registry = ToolRegistry(snapshot=world.snapshot)

registry.register(
    "my_action",
    world.my_action,
)

registry.call("my_action", value=5)

print(registry.events)

The registry records:

  • Tool name and arguments
  • Execution result or exception
  • State before and after execution
  • Whether the state changed
  • Execution duration

CLI

Run a custom scenario:

agentdrill custom --file my_test.py

The Python file must define:

  • scenario: an AgentDrill Scenario instance
  • Agent: a Python class implementing run(tools, request)

List bundled refund scenarios:

agentdrill list

Test a compatible refund agent:

agentdrill test --agent examples/my_agent.py

The generic Python API is domain-independent. The bundled refund CLI remains a domain-specific example.

Examples

The repository includes demonstrations for:

  • Refund workflows
  • CRM customer updates
  • Inventory management
  • Banking-style transfers
  • Custom business rules

Limitations

AgentDrill is currently an experimental alpha.

  • Agent code runs in the same Python process.
  • It is not a security sandbox.
  • Custom simulations and verification rules are developer-defined.
  • Tool tracing only covers calls routed through the tracing adapter.
  • Arbitrary third-party agent frameworks are not automatically supported.
  • Automatic adversarial scenario generation is not yet available.
  • Production reliability has not been established.

Do not use the current execution model to run untrusted Python code or connect tests directly to irreversible production actions.

Project Status

Version 0.7.0 — Alpha.

We are looking for developers building tool-using AI agents who want to test business-state correctness.

Feedback and reproducible failure cases are especially valuable.

License

MIT.

Metadata

Release files for agentdrill 0.7.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for agentdrill 0.7.0
File Size Uploaded Provenance Actions
agentdrill-0.7.0.tar.gz 20.7 kB With provenance Details for agentdrill-0.7.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for agentdrill 0.7.0
File Interpreter ABI Platform Actions
agentdrill-0.7.0-py3-none-any.whl Python 3 none any Details for agentdrill-0.7.0-py3-none-any.whl

Total release size: 48.1 kB

Release files / agentdrill-0.7.0.tar.gz

Download URL agentdrill-0.7.0.tar.gz
Size 20.7 kB
Tags Source
SHA-256 checksum
How to use checksums
521ee87af7035e7c1f54e12e01ebfa3376179b8e6a878d42257cd4c3cb4fb8cd
BLAKE2b-256 checksum
How to use checksums
a789dc066d1f70582f239c86d34ab0ff059ab3e64f130d09166d8140f2c3a919
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 22, 2026.

Transparency log

Release files / agentdrill-0.7.0-py3-none-any.whl

Download URL agentdrill-0.7.0-py3-none-any.whl
Size 27.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
56087b9bf34eeccdf7ee75075b1e1c97c21828554c654ace2d69c2fdb8a55b3f
BLAKE2b-256 checksum
How to use checksums
b213991d6cb5ded350e0f2a98f1aa0c7a6707b920933c42c79df62a43d6fc1cd
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 22, 2026.

Transparency log

Security for agentdrill 0.7.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for agentdrill-0.7.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 22, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.7.0 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page