Autobench

Autobench turns one-off benchmark scripts into semantic, replayable experiment evidence.

It is a YAML-first Python framework for AI and non-AI systems:

• deterministic dataset x variant execution
• sync and async application tasks
• semantic observations, checks, measurements, artifacts, and ABP traces
• built-in and custom scoring, cost derivation, policies, and paired baselines
• native Pydantic AI, OpenAI, OpenAI Agents, and HTTPX instrumentation
• explicit and automatic prompt/tool/schema/agent asset lineage
• immutable YAML records, replay, Rich reports, comparisons, and exports

Install

uv add autobench

For native SDK instrumentation:

uv add 'autobench[instrumentation]'

First Run

autobench validate examples/minimal/autobench.yaml
autobench run examples/minimal/autobench.yaml --record /tmp/autobench-minimal
autobench replay /tmp/autobench-minimal
autobench report /tmp/autobench-minimal

A task is a normal sync or async callable:

from autobench import Case, RunContext


def run(ctx: RunContext, case: Case) -> Result:
    mode = ctx.factor("mode")
    with ctx.span("subject", kind="workflow") as span:
        result = application(case.input, mode=mode)
        span.set_output(result)
        return result

The YAML spec owns reusable benchmark infrastructure: cases, variants, scoring, derivation, policies, instrumentation, and reports.

Examples

Run the offline release matrix:

make examples

Documentation

Full documentation: vcoderun.github.io/autobench

• Installation
• First Benchmark
• Use Cases
• Architecture
• YAML Spec
• Python API
• Autobench Protocol

LLM-readable indexes are available at llms.txt and llms-full.txt.

Development

uv sync --extra dev --extra instrumentation --extra openai-agents
make prod
make pre-commit

The release gate enforces Python 3.11-3.14, strict lint and typing, strict documentation builds, offline examples, and 100% source line and branch coverage.