agentic-evalkit

agentic-evalkit grades AI agent output against checks you define, and writes up the evidence behind each score.

Use it from the command line or Python to run repeatable evaluations, apply objective checks, compare results, and generate reports for review. It can evaluate Python code, command-line programs, and HTTP services without tying them to a specific AI framework.

Objective checks run before optional model-based judging. The package includes tested support for calibrated judge evidence while leaving model-provider selection to the caller. The built-in reference judge always returns the same verdict for the same input, and it's advisory only; it cannot approve a release.

Why not promptfoo, Inspect, DeepEval, Braintrust, or LangSmith? Those tools handle the workflow side of evals — running lots of prompts, wiring checks into CI, tracking experiments over time. Use one of them for that. This package solves a narrower, stricter problem: trusting the result you get.

• An AI judge has to prove it agrees with real human-labeled answers before it's allowed to approve anything
• Two runs are only compared once we can prove they ran under matching conditions
• A bug in your own code is never confused with the AI simply getting the answer wrong
• A fuzzy AI opinion can never override a hard requirement like "the code must compile"
• On grounded-citation tasks, a planted marker echoed back in an answer is caught before it inflates a score

None of the tools above solve these problems. Full comparison and reasoning: docs/prior-art.md.

Start with the quickstart guide. Design boundaries: architecture specification.

Quickstart

pip install agentic-evalkit
agentic-evalkit doctor
agentic-evalkit init --preset gsm8k --output eval.yaml
agentic-evalkit run eval.yaml --limit 5 --yes

This resolves the curated GSM8K preset from Hugging Face, runs five samples through the packaged smoke target, grades them with a normalized exact-match grader, and writes a canonical JSON report. No importer code, manual dataset download, datasets, pyarrow, or Docker is required. See docs/guides/quickstart.md for the full walkthrough, including the standalone report command that regenerates a self-contained HTML report from that JSON.

Python API

The CLI is built on a small, curated Python API for integrations that need more than the built-in presets: wrap your own system as a target, then describe a run with a typed manifest.

from agentic_evalkit import CallableTarget, DatasetRef, EvalRunManifest, EvalRunner

def my_system(sample_input: dict) -> dict:
    return {"answer": solve(sample_input["question"])}

target = CallableTarget(my_system, name="my-system")
manifest = EvalRunManifest(
    run_name="quickstart", adapter="gsm8k@1", grader="normalized-exact@1",
    target_name="my-system", dataset_ref=DatasetRef(provider="huggingface", dataset_id="openai/gsm8k"),
)

EvalRunner(...).run(manifest) then drives dataset resolution, execution, and grading end to end. CallableTarget satisfies ExecutionTarget — agentic-evalkit's only system-under-test boundary — which is also exported at the top level for anyone implementing a custom target. Everything else — additional targets, graders, reporters, dataset providers, benchmark adapters, and statistics — is one import away under its own subpackage (agentic_evalkit.graders, agentic_evalkit.reporters, and so on); see the HTTP agent example for a complete, runnable Python-API script with the catalog/adapter/grader/ artifact-store wiring this snippet omits for brevity.

Optional extras

The swebench extra (pip install agentic-evalkit[swebench], pulling in swebench>=4.1,<5 and docker>=7.1,<8) is the only extra agentic-evalkit declares. It backs SweBenchDockerHarnessExecutor, the container-based SWE-bench Verified harness executor landed in ADR-0014: with the extra installed and a reachable Docker daemon, swebench-harness@1 grades a real resolved/unresolved verdict instead of reporting unavailable. The base install still ships without Docker or any model-provider SDK — see ADR-0009 for the extras policy.

Documentation

• Quickstart — install to first report
• CLI reference — commands, options, offline behavior, and exit codes
• Providers — local formats, Hugging Face auth, cache/offline
• Graders — objective-first order, hard gates, calibrated judges
• Targets — callable, subprocess, HTTP, and MCP-stdio execution targets
• SWE-bench — preview/prediction workflow and the harness boundary
• HTTP agent example — evaluating a real HTTP agent endpoint

Repository boundary

This project imports no host-repo internals — systems are reached only through the public ExecutionTarget protocol (callable, subprocess, HTTP, or MCP-stdio adapters); see ADR-0001 and ADR-0006.