agentic-evalkit 0.2.0

Evaluate agentic systems with reproducible, evidence-first grading.

agentic-evalkit

agentic-evalkit is a standalone evaluation toolkit for agentic systems. It combines dynamic dataset discovery, typed evaluation contracts, benchmark-valid grading, calibrated judges, statistical reporting, and a developer-friendly Python API and CLI.

agentic-evalkit separates datasets, grading, and reporting from the system under test through callable/subprocess/HTTP targets, and objective checks gate before judges. Existing evaluation frameworks couple dataset access, grading, and reporting to specific agent platforms or model-provider SDKs; this package's neutral ExecutionTarget protocol makes any callable, subprocess, or HTTP system evaluable without framework lock-in.

Coexistence note: legacy evaluation code may remain in host repositories. This package neither imports nor migrates it — integration happens only through the public ExecutionTarget protocol described above.

See the architecture specification for the full design, or jump straight to the quickstart guide.

Identity

• Distribution and repository: agentic-evalkit
• Python package: agentic_evalkit
• CLI: agentic-evalkit

Quickstart

pip install agentic-evalkit
agentic-evalkit doctor
agentic-evalkit init --preset gsm8k --output eval.yaml
agentic-evalkit run eval.yaml --limit 5 --yes

This resolves the curated GSM8K preset from Hugging Face, runs five samples through the packaged smoke target, grades them with a normalized exact-match grader, and writes a canonical JSON report. No importer code, manual dataset download, datasets, pyarrow, or Docker is required. See docs/guides/quickstart.md for the full walkthrough, including the standalone report command that regenerates a self-contained HTML report from that JSON.

Python API

The CLI is built on a small, curated Python API for integrations that need more than the built-in presets: wrap your own system as a target, then describe a run with a typed manifest.

from agentic_evalkit import CallableTarget, DatasetRef, EvalRunManifest, EvalRunner

def my_system(sample_input: dict) -> dict:
    return {"answer": solve(sample_input["question"])}

target = CallableTarget(my_system, name="my-system")
manifest = EvalRunManifest(
    run_name="quickstart", adapter="gsm8k@1", grader="normalized-exact@1",
    target_name="my-system", dataset_ref=DatasetRef(provider="huggingface", dataset_id="openai/gsm8k"),
)

EvalRunner(...).run(manifest) then drives dataset resolution, execution, and grading end to end. CallableTarget satisfies ExecutionTarget — agentic-evalkit's only system-under-test boundary — which is also exported at the top level for anyone implementing a custom target. Everything else — additional targets, graders, reporters, dataset providers, benchmark adapters, and statistics — is one import away under its own subpackage (agentic_evalkit.graders, agentic_evalkit.reporters, and so on); see the HTTP agent example for a complete, runnable Python-API script with the catalog/adapter/grader/ artifact-store wiring this snippet omits for brevity.

Optional extras

The parquet, judges, and swebench extras (e.g. pip install agentic-evalkit[swebench]) are currently reserved placeholders per ADR-0009: installing them adds nothing today. They reserve the capability names that agentic-evalkit doctor reports on, ahead of their own future implementation.

Documentation

• Quickstart — install to first report
• CLI reference — commands, options, offline behavior, and exit codes
• Providers — local formats, Hugging Face auth, cache/offline, plugins
• Graders — objective-first order, hard gates, calibrated judges
• Targets — callable, subprocess, and HTTP execution targets
• SWE-bench — preview/prediction workflow and the harness boundary
• HTTP agent example — evaluating a real HTTP agent endpoint

Repository boundary

This project does not modify or import Agentic Runtime Platform or ExecutionKit internals. Those systems may be evaluated through stable callable, subprocess, or HTTP target adapters — see ADR-0001 and ADR-0006.