aieval

Deterministic-first runtime validation and reliability toolkit for LLM outputs and agent actions.

Catch predictable failures (schema violations, missing required keys, JSON format errors, invalid tool call args, policy constraints) in sub-millisecond execution before spending latency and money on semantic LLM-as-a-judge scoring.

Key Highlights

• ⚡ Deterministic First: Cheap, fast assertions run before calling model judges.
• 📦 Zero Bloat: Minimal external footprint (pydantic>=2.0.0 as schema engine, pure Python standard library for the rest).
• 🔄 Dual Sync & Async: Native support for synchronous scripts (evaluate()) and asynchronous pipelines (await aevaluate()).
• 🤖 Agent Tool Call Pre-flight: Validates function names, required arguments, type safety, and stringified JSON (OpenAI tool call format).
• 🎚️ Severity Routing: Flexible threshold routing (fail_on="error" vs strict fail_on="warning").
• 🧪 CLI & Golden Fixtures: Run validation from the command line on JSON test fixtures (aieval run <fixture.json>).
• 🎯 Full TypeScript Parity: Shares the exact failure taxonomy, code conventions, and golden test formats with @hamza1331/aieval.

Installation

pip install aieval-py

Quickstart

1. Synchronous Output Evaluation

from aieval import evaluate, schema_check, required_fields
from pydantic import BaseModel, Field

class AnalysisReport(BaseModel):
    summary: str
    confidence_score: float = Field(ge=0.0, le=1.0)
    tags: list[str]

llm_output = '{"summary": "All systems nominal", "confidence_score": 0.95, "tags": ["ops", "prod"]}'

result = evaluate(
    output=llm_output,
    checks=[
        schema_check(AnalysisReport),
        required_fields(["summary", "confidence_score"]),
    ],
)

if result.passed:
    print(f"Passed in {result.summary.duration_ms}ms!")
else:
    for failure in result.failures:
        print(f"[{failure.code}] {failure.message} (path: {failure.path})")

2. Pre-flight Agent Tool Call Validation

Validate tool calls before execution to avoid runtime exceptions and agent failure loops:

from aieval import evaluate, tool_call_check
from pydantic import BaseModel

class SendEmailArgs(BaseModel):
    recipient: str
    subject: str
    body: str

# Works with both native dicts and OpenAI stringified JSON arguments:
raw_tool_call = {
    "name": "send_email",
    "arguments": '{"recipient": "team@example.com", "subject": "Update"}'
}

result = evaluate(
    output=raw_tool_call,
    checks=[
        tool_call_check("send_email", schema=SendEmailArgs, required_args=["body"]),
    ]
)

print(result.passed) # False - missing required argument 'body'

3. Asynchronous Pipeline (aevaluate)

import asyncio
from aieval import aevaluate, valid_json, regex_match

async def main():
    result = await aevaluate(
        output="Order #12345 confirmed.",
        checks=[
            regex_match(r"Order #\d+"),
        ],
    )
    print("Passed:", result.passed)

asyncio.run(main())

CLI Usage

Run Golden Test Fixtures

aieval run path/to/fixture.json

Or output raw JSON:

aieval run path/to/fixture.json --json

Check Arbitrary Files

aieval check output.json --specs checks.json

Failure Code Taxonomy

Examples

Runnable walkthrough scripts are available in the examples/ directory:

• 01_basic_llm_output.py: Schema and field constraint validation on LLM output.
• 02_agent_tool_guardrail.py: Agent tool call pre-flight validation (OpenAI stringified arguments & multi-tool selection).
• 03_async_and_severity_routing.py: Asynchronous concurrent pipeline with severity routing.
• 04_semantic_judge_adapter.py: Integrating advisory LLM-as-a-judge checks via SemanticJudge.

License

MIT