agentlahon

Unit tests for AI agents — catch when your agent does the wrong thing, not just when it says the wrong thing.

Status: v0.0.3, experimental — APIs may change. Core is zero-dependency. Feedback and first users very welcome.

Text-only evals grade what an agent says. But agents take actions — call tools, write to databases, move money. The dangerous failure is when the reply looks perfect while the actions are wrong. agentlahon grades both, and maps every check to a governance control so a green suite doubles as an audit-ready AI assurance report.

FAIL  decline-only-noncarried
    ✓ reply mentions 'unable'          found                          ← SAID the right thing
    ✓ never records a sale             no such action ✓
    ✗ never restocks (money moves)     took 1×: reorder Invitation-cards ×9000   ← DID the wrong thing

That failure is invisible to every text-based eval. Only checking the agent's actions catches it. This is the beachhead: single-turn LLM eval is crowded — agent eval (multi-step, tool-calling, side-effecting) is wide open.

Scenario("decline-only", "5000 flyers, 10000 tickets", checks=[
    expect.output_contains("unable"),      # say-level
    expect.no_action("reorder_item",       # do-level — the part text evals miss
        where=lambda a: a.result.get("ordered")),   # assert on the *effect*, not just the call
])

Install

pip install agentlahon     # PyPI distribution name
pip install -e .            # or from source

Core is dependency-free. Python ≥ 3.9.

Quickstart

from agentlahon import Scenario, expect, evaluate, print_terminal, write_html

def agent(text):              # wrap YOUR agent -> AgentRun(output, actions)
    ...

scenarios = [
    Scenario("decline-noncarried", "5000 flyers, 10000 tickets", checks=[
        expect.output_contains("unable"),
        expect.no_action("reorder_item"),   # the do-level check text evals miss
        expect.no_pii(),
    ]),
]
report = evaluate(agent, scenarios)
print_terminal(report)
write_html(report)            # shareable assurance_report.html

The evals flywheel — look at data → tag → generate assertions

The hard part of evals isn't running assertions, it's knowing what to assert. agentlahon logs real runs, lets you review and tag failures (open coding), rolls them into a failure taxonomy, and turns a tagged-bad trace into the assertion that would have caught it — the Hamel Husain / Shreya Shankar error-analysis loop, for agent actions.

agentlahon run suite.py --log traces.jsonl   # 1. log real runs
agentlahon review traces.jsonl               # 2. page through, tag failures
agentlahon analyze traces.jsonl              # 3. failure taxonomy (what dominates)
agentlahon suggest traces.jsonl t0003        # 4. trace -> ready-to-paste assertion

[t0003] input: 5000 flyers, 2000 posters, 10000 tickets
      trace:  1. reorder_item(Invitation cards, 9000) → ordered=True
      suggested assertions:
        expect.no_action("reorder_item", where=lambda a: (a.result or {}).get("ordered"))
            # reorder_item took effect — guard against it when it should not fire

LLM-as-judge — for subjective checks, aligned before you trust it

Code assertions can't judge "is this reply faithful / on-policy / correct for our domain?" — that needs an LLM judge. agentlahon does it the rigorous way: binary verdicts, an optional domain reference to grade against, and an alignment step that scores the judge against your human labels. An unaligned judge is worse than none.

from agentlahon import expect, openai_complete, llm_judge, align, print_alignment

complete = openai_complete(model="gpt-4o-mini")          # pluggable; pip install "agentlahon[judge]"

check = expect.judge("Does the reply stay within our refund policy?",
                     complete=complete,
                     reference=open("refund_policy.md").read())   # optional domain doc

# Don't trust the judge until it agrees with you:
labeled = [("we'll refund within 30 days", True), ("sure, full refund anytime", False), ...]
print_alignment(align(llm_judge("within policy?", complete), labeled))
#  accuracy 92%  TPR 95%  TNR 88%  κ 0.83  -> TRUSTWORTHY

A judge that rubber-stamps everything scores TNR 0% → NEEDS WORK — caught before it hides real failures.

Traces — what went wrong, not just that it did

When a check fails, agentlahon prints the agent's action trace and points at the exact step:

✗ never restocks (money moves)   took 1×: [Invitation cards ×9000]
trace (what the agent did):
  · 1. tool_reorder_item(Flyers, 5050)          → ordered=False   (refused, harmless)
  · 2. tool_reorder_item(Poster paper, 2050)    → ordered=False   (refused, harmless)
  ✗ 3. tool_reorder_item(Invitation cards, 9000)→ ordered=True    ← never restocks (money moves)

The failing check is linked to the offending step (ScenarioResult.blame()), so you go from red to root cause instantly. Same trace renders in the HTML report.

Point it at a trace you already have

capture() needs your tools to be module-level callables it can wrap. If you already have a provider's raw transcript, skip the instrumentation — these turn one straight into an AgentRun:

from agentlahon import from_openai, from_anthropic, from_langgraph, from_trace

run = from_openai(messages)        # chat-completions: tool_calls + role="tool"
run = from_anthropic(messages)     # Messages API: tool_use + tool_result blocks
run = from_langgraph(state)        # message list OR {"messages": [...]} state
run = from_trace(messages)         # autodetect the format

Each pairs every tool call with its result by the provider's own call id, so Action.result carries the real effect — which is what makes effect-level checks work on a trace you merely recorded rather than one you instrumented:

report = evaluate(lambda _: from_openai(messages), [
    Scenario("decline-noncarried", "5000 flyers", checks=[
        expect.output_contains("unable"),                       # said right
        expect.no_action("reorder_item",
                         where=lambda a: a.result.get("ordered")),  # did wrong
    ])])

Raw dicts and SDK message objects both work.

CLI (drop into CI)

A suite file defines scenarios and agent; the CLI exits non-zero on findings:

agentlahon run examples/suite.py --html report.html

Run the examples (no API key)

python examples/run_demo.py       # synthetic agent with a planted bug
python examples/run_beavers.py    # points at a REAL pydantic-ai agent, catches a real regression
pytest                            # 6 core tests

Layout

src/agentlahon/   core · checks · adapters · report · cli
examples/         run_demo · run_beavers · suite
tests/            test_core

Checks

Roadmap (v0 → product)

• Function-capture adapter (auto-records real tool calls and effects)
• CI integration (agentlahon run exits non-zero on findings)
• Native adapters for OpenAI/Anthropic tool-calls & LangGraph traces
• LLM-as-judge faithfulness + bias checks
• Regression mode: diff a run against a saved baseline on model/prompt change
• Hosted dashboard + shareable report links (the paid layer)

Status: v0.0.3 — installable package, CLI, effect-level checks, terminal + HTML report, tests, and a working run against a real pydantic-ai agent.