agentsproof

Drop the SDK into your Python agent, define what "good" means, and get a shareable proof report.

Install

pip install agentsproof

Quick start — single run (sync)

Works with any Python agent — OpenAI, Anthropic, LangChain, CrewAI, or plain functions.

import os
from agentsproof import AgentsProof

ap = AgentsProof(api_key=os.environ["AGENTSPROOF_API_KEY"])

def run_my_agent(user_query: str):
    run = ap.start_run(
        project_slug="my-coding-agent",
        label="Answer coding question",
        input={"query": user_query},
        goal="Search the web for relevant docs and return a working code solution",
    )

    plan = run.trace("llm_call", "gpt-4o", lambda: openai_call(user_query), input=user_query)
    results = run.trace("tool_call", "web_search", lambda: web_search(plan))
    final_answer = run.trace("llm_call", "gpt-4o", lambda: openai_call(results))

    result = run.complete({"answer": final_answer})
    print(f"Report: {result['publicUrl']}")
    # → https://agentsproof.dev/r/abc123

Quick start — async agent

import asyncio, os
from agentsproof import AgentsProof

ap = AgentsProof(api_key=os.environ["AGENTSPROOF_API_KEY"])

async def run_my_agent(user_query: str):
    run = ap.start_run(
        project_slug="my-coding-agent",
        input={"query": user_query},
        goal="Return a working code solution",
    )

    plan = await run.atrace("llm_call", "gpt-4o", lambda: async_openai_call(user_query))
    results = await run.atrace("tool_call", "web_search", lambda: async_web_search(plan))
    final_answer = await run.atrace("llm_call", "gpt-4o", lambda: async_openai_call(results))

    result = await run.acomplete({"answer": final_answer})
    print(f"Report: {result['publicUrl']}")

asyncio.run(run_my_agent("How do I reverse a list in Python?"))

Run against an existing Golden

Pass golden_id to use a Golden as the eval context. The Golden's input, goal, and expected_output fill in automatically as defaults — any field you also provide explicitly takes precedence. The Golden's success_criteria, trace_assertions, failure_modes, and expected_behavior are applied at grading time with no extra work.

run = ap.start_run(
    project_slug="my-coding-agent",
    golden_id="abc-123",           # all Golden context loaded automatically
    label="Ad-hoc run against Golden",
    # input is optional — auto-filled from the Golden when omitted
)

result = my_agent(run)
run.complete(result)

Inline eval fields — no Golden required

Supply the full eval context directly to start_run() without creating a Golden in the dashboard. Useful for one-off runs or CI scripts.

run = ap.start_run(
    project_slug="my-coding-agent",
    input={"query": user_query},
    goal="Return a working TypeScript solution.",
    success_criteria=[
        "Returns syntactically valid TypeScript",
        "Handles the null / empty-array case",
    ],
    trace_assertions=["max_steps:5", "must_call:web_search"],
    failure_modes=["hallucinated_api", "missing_null_check"],
    expected_behavior="Agent searches docs, then writes a solution with a null guard.",
)

Proof Suites — regression testing

import os
from agentsproof import AgentsProof

ap = AgentsProof(api_key=os.environ["AGENTSPROOF_API_KEY"])

def handler(input, ctx):
    run = ctx.start_run()
    result = my_agent(input)
    run.complete({"answer": result})

result = ap.run_proof_suite(
    project_slug="my-coding-agent",
    suite_slug="core-behaviors",
    handler=handler,
)
print(result)
# → {"passedCases": 17, "failedCases": 1, "overallScore": 0.91, "publicUrl": "..."}

Async proof suite

async def async_handler(input, ctx):
    run = ctx.start_run()
    result = await my_async_agent(input)
    await run.acomplete({"answer": result})

result = await ap.arun_proof_suite(
    project_slug="my-coding-agent",
    suite_slug="core-behaviors",
    handler=async_handler,
)

---

API

AgentsProof(api_key, base_url?)

Create a client. Get your API key from agentsproof.dev.

client.start_run(...) → AgentRun

Param	Type	Required	Description
project_slug	str	yes	Your project identifier
input	Any	yes¹	The initial input or prompt to the agent
golden_id	str	no	ID of an existing Golden. Loads its input, goal, expected_output, success_criteria, trace_assertions, failure_modes, and expected_behavior automatically. Explicit params override Golden defaults.
label	str	no	Human-readable label shown in the dashboard run list
goal	str	no	What this run should accomplish. Drives goal_completion scoring.
expected_output	Any	no	Reference output. Grader compares actual output against this for output_quality scoring.
expected_behavior	str	no	Step-by-step description of a correct execution. Informs all 5 grading axes.
success_criteria	list[str]	no	Explicit checklist evaluated one-for-one in criteria_results. Overrides LLM-inferred criteria from goal.
trace_assertions	list[str]	no	Deterministic assertions: must_call:<name>, must_not_call:<name>, max_steps:<n>, min_steps:<n>. Free-text entries are sent to the LLM grader as extra criteria.
failure_modes	list[str]	no	Known bad outcomes. Grader penalises runs where these are observed.
metadata	dict	no	Arbitrary key/value pairs for filtering and grouping in the dashboard.

¹ input is required unless golden_id is provided, in which case the Golden's input is used as the default.

run.trace(type, name, fn, input?, extract?) → T

Wrap a sync callable and auto-log it as a step with latency captured.

run.atrace(type, name, fn, input?, extract?) → Awaitable[T]

Wrap a sync or async callable. Use in async agent code.

Token count and cost are captured automatically in priority order:

1. extract — your own callable, receives the step output, returns {"token_count": int, "cost_usd": float} (both optional).
2. Auto-detection — the SDK sniffs output.usage (or output["usage"]) for Anthropic (input_tokens + output_tokens) and OpenAI-compatible (total_tokens or prompt_tokens + completion_tokens) shapes.
3. null — if neither works, both fields are omitted.

# Anthropic / OpenAI — auto-detected, no extra code needed
result = run.trace("llm_call", "claude", lambda: anthropic_call(prompt))

# Any other provider — supply an extractor
result = run.trace(
    "llm_call", "my-model",
    lambda: call_my_llm(prompt),
    input=prompt,
    extract=lambda out: {"token_count": out.usage.tokens, "cost_usd": out.billed_usd},
)

run.log_step(payload)

Manually log a step without wrapping a function. Step types: llm_call | tool_call | tool_result | memory_read | memory_write.

run.complete(output) → {"publicUrl": str}

Finish the run, trigger grading, and get back the public report URL.

run.acomplete(output) → Awaitable[{"publicUrl": str}]

Async version of complete().

client.run_proof_suite(...) / client.arun_proof_suite(...)

Run approved Goldens locally against your agent. AgentsProof never executes user code remotely.

The SDK never raises on logging failures — steps are fire-and-forget so the SDK cannot crash your agent.

---

How grading works

Each run is automatically scored on 5 axes:

Axis	Weight	What it measures
Goal completion	35%	Did the agent achieve the stated goal?
Output quality	20%	Is the final output correct and complete?
Tool accuracy	20%	Were tool calls well-formed and necessary?
Step efficiency	15%	Did it avoid redundant steps or loops?
Safety	10%	Did it avoid unsafe or off-policy actions?

Weights adjust automatically — if your agent makes no tool calls, tool_accuracy weight is redistributed to goal_completion and output_quality.

criteria_results — how the checklist is populated:

What you provide	What the grader receives	Result
success_criteria=[...] (from Golden or directly)	Explicit bullet list	One pass/fail entry per criterion
goal only (no success_criteria)	Free-text goal prose	Grader infers its own criteria from the goal text
Neither	Nothing	criteria_results is empty

trace_assertions — structured patterns (must_call:*, max_steps:*, etc.) are evaluated deterministically before the LLM runs. Free-text entries are passed to the LLM grader as additional criteria.

Providing a goal always improves accuracy. Without it, the grader infers intent from the raw input alone.

Every report includes per-axis reasoning text and a criteria_results checklist so the score is always explainable.