ageval

English · 简体中文

How to avoid rewriting scaffolds for the huge set of agent runtime × model × environment combinations?

ageval keeps one stable Core and swaps the agent under test and the environment through plugins. Install the CLI and skills so a coding agent can design or convert a benchmark and finish the eval; after results land on the Hub, datasets, plugins, and Agent packages can be shared or reused publicly.

Getting started

uv tool install ageval-cli
# install everything, or only what you need
uv tool install 'ageval-cli[all]' # every CLI extra (not the Hub service)
uv tool install 'ageval-cli[e2b]' # one extra at a time

ageval -V

Run a dataset straight from the Hub (ageval.zjureal.com; the CLI default), or any local dataset root:

ageval registry list                        # datasets visible on the Hub
ageval run official/minimal-demo@0.1.3 --task terminal-jsonl-agg
ageval run <org>/<name>@<version> --task <task-id>
ageval executors -v
ageval view <org>/<name>@<version> --no-browser

Install skills

Install skills for your local coding agent (CLI usage, plugins, dataset authoring, and more):

# install all
npx skills add ZJU-REAL/ageval
# install specific ones
npx skills add ZJU-REAL/ageval --skill ageval-cli

Develop from source

To try the in-repo dataset examples and Agent catalog packs, or to build from source, clone the repo:

git clone https://github.com/ZJU-REAL/ageval.git
cd ageval
uv sync --frozen --all-packages
uv run ageval -V

Run the in-repo minimal example and inspect the results in the local Viewer:

uv run ageval tasks examples/datasets/minimal-demo
uv run ageval run  examples/datasets/minimal-demo --task terminal-jsonl-agg
uv run ageval view examples/datasets/minimal-demo --no-browser

Features

Quickly switch the agent under test

Environments and agent runtimes both plug in. The default path is ACP; nooa, dsh, and miniswe use the same plugin path. Change one line in profiles.yaml, or pass --agent for a one-off switch.

Let the Agent run the eval

With the CLI and skills installed, a coding agent can author or convert a dataset and run the eval end to end. Afterwards, ageval view replays the trajectory locally: phase timing, tool calls, and a reproduce command for failed tasks. See Getting started.

Two scoring styles

Only a separate evaluator.py can return PASS:

• Default is a deterministic script: compare artifacts to gold, optionally split into checks
• Open-ended tasks can add a judge role on the same profiles.yaml and use LLM-as-judge.

Share and reuse on Hub

Upload datasets, plugins, Agent packages, and results to ageval Hub. Leaderboard scores name the Agent and environment used; pull a published Agent with --agent; compare models side by side.

Screenshots

Leaderboard: scores by Agent and model

Trajectory: replay tool calls in the local viewer

How it works

End-to-end flow

1. ageval lock resolves the plugin graph (ExtensionGraph), checks capabilities and credentials, and writes lock.json (secrets stay locators).
2. ageval run opens an environment and uploads task files (local / Docker / E2B, …).
3. run.py drives the task loop inside that environment; swapping env or Agent does not require editing this file.
4. Only evaluator.py can return PASS; gold uploads after the run; cleanup always runs.

%%{init: {"theme":"base","themeVariables":{"actorBkg":"#eaf1ff","actorBorder":"#5B7BFF","actorTextColor":"#10233f","actorLineColor":"rgba(45,49,66,0.2)","signalColor":"#4f5d75","signalTextColor":"#2d3142","labelBoxBkgColor":"#eaf1ff","labelBoxBorderColor":"#5B7BFF","labelTextColor":"#10233f","noteBkgColor":"#f1f5f9","noteBorderColor":"#64748b","noteTextColor":"#1e293b"}}}%%
sequenceDiagram
    autonumber
    actor u as you
    participant r as run entry
    participant e as Environment<br/>local / docker / e2b…
    participant t as run.py + Agent
    participant v as evaluator.py

    u->>r: ageval lock<br/>dataset + profiles → lock.json
    Note over r: the check fails — the run does not start
    u->>r: ageval run
    r->>r: mint run identity
    r->>e: open one environment
    r->>e: upload task files
    e->>t: execute run.py
    loop task loop
        t->>t: ACP invoke · attach_stdio
    end
    t-->>r: trajectory.jsonl
    Note over r,v: run ends → gold uploads, then evaluation
    r->>v: run evaluator.py
    v-->>r: PASS / FAIL / ERROR
    r->>r: record · finally cleanup<br/>lock.json · result.json · trajectory.jsonl

Core base

ageval Core is a fixed five-phase pipeline: lock → environment → run → evaluate → record. Inputs are the user, dataset, and profiles; outputs land in evidence (lock.json, result.json, trajectory.jsonl). At ageval lock, one environment plugin and one Agent plugin are bound. Before the run starts, limits cap wall-clock time, memory, process forks, and call counts; cleanup always runs. Swapping an Agent or an environment does not require changing Core.

Plugins

Core does not hard-code a particular Agent or environment. Plugins declare what they provide and what capabilities they need; ageval lock resolves a dependency graph (ExtensionGraph), and later dispatch follows that graph.

# Agent plugin (e.g. dsh)
plugin_id: dsh
slots:
  exclusive:
    - id: executor
inject:
  - service: environment
    capabilities: [exec, upload]

# Environment plugin (e.g. docker)
plugin_id: docker
slots:
  exclusive:
    - id: environment

Project structure

ageval/
├── src/ageval/
│   ├── cli/                         # argv, help, exit code
│   ├── application/
│   │   ├── composition.py           # sole production wiring; CLI imports build_* here
│   │   ├── lock.py                  # load_and_lock
│   │   ├── run.py                   # mint identity → run_attempt
│   │   ├── campaign.py / suite/     # matrix · suite · Always-k
│   │   └── agent_ops/ / plugin_ops / registry_ops/
│   ├── attempt/                     # Attempt pipeline
│   │   ├── __init__.py              # run_attempt
│   │   └── phases/                  # environment → run → evaluate → record · cleanup
│   ├── config/                      # dataset + task.yaml + profiles
│   ├── environments/protocol.py     # EnvironmentProvider · caps; no vendor SDK
│   ├── plugins/
│   │   ├── slots.py                 # exclusive / chain
│   │   └── contrib/                 # acp · local · docker/attempt (official base, in the wheel) · e2b · daytona · ssh
│   ├── runtime/                     # identity, parent Agent Service, task_worker
│   ├── evaluation/                  # bind PASS
│   └── evidence/                    # trajectory.jsonl layout
├── src/ageval_sdk/                  # ageval_sdk for run.py (no PASS, no host credentials)
├── plugins/                         # external ageval.plugin/1 (nooa, dsh, miniswe, …)
├── examples/
│   ├── datasets/
│   │   ├── minimal-demo/            # terminal-jsonl-agg · tau2-dialog-min · multiagent-env-min
│   │   └── tau3-airline-5/            # airline-00 … airline-04
│   └── agents/                      # ageval.agent/1
├── apps/shared                      # Hub + Viewer overlap chrome
├── apps/viewer                      # ageval view SPA
├── apps/hub                         # Hub SPA
├── services/registry/               # package + results HTTP
├── docs/                            # mechanism design
└── website/                         # product docs

Docs

• Usage: docs site (source)
• Design: docs/
• Examples: examples/README.md
• AGENTS.md
• ARCHITECTURE.md