AgentProctor

Description

The AgentProctor module is a testing and evaluation library designed for AI agents, enabling the testing of various behaviors and properties that cannot be effectively evaluated using standard regression or unit tests. This library provides a structured approach to evaluate AI responses against defined criteria, ensuring that agents behave as expected in diverse scenarios.

Why AgentProctor?

Traditional testing breaks down when applied to LLM-based systems.

AgentProctor provides a structured way to:

• Evaluate non-deterministic outputs
• Define reusable evaluation criteria
• Measure response quality with explainability
• Build confidence in AI systems before production

Documentation

• README - Project overview and features
• Quickstart - Get started in 5 minutes
• Changelog - Version history and updates
• License - MIT License

Features

• LLM-native evaluation: Designed specifically for testing AI-generated outputs
• Schema Validation: Utilizes Pydantic for robust validation of input data structures.
• Custom Evaluators: Supports the creation of custom evaluators to assess agent responses based on specific criteria.
• Detailed Reporting: Generates structured output that includes evaluation results, reasons for success or failure, and confidence levels.
• Extensible Architecture: Easily extendable to accommodate new evaluation criteria or agent behaviors.

Tech Stack

• Python: Version 3.10 or higher.
• LiteLLM: For lightweight language model interactions.
• Pydantic: For data validation and settings management.
• UV: For dependencies management.
• pytest: For testing framework.
• Ruff: For linting and formatting.

Installation

To install the AgentProctor library, you can use pip. Make sure you have Python 3.10 or higher installed.

pip install agentproctor

This command installs the library along with its development dependencies.

Usage

To use the AgentProctor library, first import the necessary classes from the package. Below is a simple example of how to create an evaluation request and evaluate it using a custom evaluator.

Since AgentProctor uses LiteLLM to handle the LLM API calls, you must set your prefered LLM API key as an environment variable.

Fixed-type evaluator

from agentproctor.schemas import EvaluationRequest
from agentproctor.base_evaluator import BaseEvaluator

# Create an evaluation request
request = EvaluationRequest(
    response="Your agent's response here",
    model="gpt-4o-mini",
    additional_context="Optional context to help the evaluator"
)

# Create a custom evaluator
class MyEvaluator(BaseEvaluator):
    @property
    def criteria(self):
        return "Your evaluation criteria here."

# Evaluate the request
evaluator = MyEvaluator()
result = evaluator.evaluate(request)

print(result)

>>> EvaluationResult(
...     passed=False,
...     reason="A rejection reason.",
...     confidence=0.75,
>>> )

Dynamic type evaluator

from agentproctor.schemas import EvaluationRequest
from agentproctor.base_evaluator import BaseEvaluator

# Create a custom evaluator
class EntityReferenceEvaluator(BaseEvaluator):
    @property
    def criteria(self):
        """
        Defines the evaluation criteria dynamically using parameters provided at initialization.

        The BaseEvaluator accepts arbitrary keyword arguments (`kwargs`) that can be used
        to customize evaluation logic at runtime. This enables reusable evaluators that
        adapt to different inputs without redefining new classes.

        For example, this evaluator checks whether a specific entity is referenced in
        the response by accessing `self.kwargs["entity"]`.
        """
        return f"The user is referencing {self.kwargs["entity"]}"

# Create an evaluation request
request = EvaluationRequest(
    response="I'm speaking about Lionel Messi.",
    model="gpt-4o-mini",
)

# Evaluate the request
evaluator = EntityReferenceEvaluator(entity="Lionel Messi")
result = evaluator.evaluate(request)

print(result)

>>> EvaluationResult(
...     passed=True,
...     reason="The user is clearly speaking about Lionel Messi",
...     confidence=1.0,
>>> )

File Structure Overview

.
├── docs
│   ├── LICENSE.md
│   ├── QUICKSTART.md
│   └── README.md
├── notebooks
│   └── use_examples.ipynb
├── src
│   └── agentproctor
│       ├── __init__.py
│       ├── base_evaluator.py
│       ├── schemas.py
│       └── evaluators
│           ├── __init__.py
│           ├── entity_reference.py
│           └── language.py
├── tests
│   ├── conftest.py
│   ├── test_base_evaluator.py
│   ├── TEST_CATALOG.md
│   ├── test_entity_evaluator.py
│   ├── test_evaluation_request.py
│   └── test_language_evaluator.py
├── .gitignore.toml
├── .pre-commit-config.toml
├── pyproject.toml
└── uv.lock

• docs: Contains the library documentation.
• notebooks: Notebooks with worked examples about how to use this module.
• src/agentproctor/: Contains the main library code.
• tests/: Contains unit tests for the library components.
• pyproject.toml: Project metadata and dependencies.

Design Philosophy

AgentProctor is built around a few core principles:

• Evaluation > Assertion: AI systems require qualitative validation, not just equality checks
• Explainability matters: Every evaluation should provide reasoning, not just a result.
• Composable abstractions: Evaluators should be reusable and easy to extend.
• Developer experience first: Minimal boilerplate, maximum clarity.

Contributing Guidelines

We welcome contributions to improve the AgentProctor library! Here’s how you can help:

1. Fork the repository: Click the "Fork" button on the top right of the repository page.
2. Create a new branch: Use a descriptive name for your branch.

   git checkout -b feature/your-feature-name
   

3. Make your changes: Implement your feature or fix.
4. Write tests: Ensure that your changes are covered by tests.
5. Commit your changes: Write clear and concise commit messages.

   git commit -m "Add feature: your feature description"
   

6. Push to your branch:

   git push origin feature/your-feature-name
   

7. Create a pull request: Go to the original repository and create a pull request.

❤️ Support the Project

If you find AgentProctor useful, consider supporting its development.

Your support helps:

• Maintain and improve the library
• Add new evaluators and features
• Keep the project actively maintained

👉 Sponsor here: https://github.com/sponsors/diego-mourino

👋 About Me

Hi, I'm Diego — an AI Engineer focused on building reliable systems around LLMs and AI agents.

This project is part of my effort to:

• Explore better ways to evaluate AI systems
• Share practical tools with the community
• Demonstrate production-grade engineering practices

📬 Contact

💼 LinkedIn: https://www.linkedin.com/in/diego-mourino/?locale=en-US

📧 Email: diego.mourino.rs@gmail.com

If you're interested in collaboration, consulting, or have feedback, feel free to reach out.