evalarc 0.15.0

Find the agent regression behind a better score.
Review changed checks, follow the recorded actions, and hand off evidence someone else can verify.

Try the recorded failure → · First local review · Hugging Face · 简体中文

90% → 93.75%. Two checks improve. One previously passing check fails. A tool commits a note but returns an error. Retrying with a new key writes the note again. EvalArc exposes that regression instead of letting the higher average score settle the review.

EvalArc is a Python command-line reviewer for agent evaluations. It compares saved runs check by check, applies explicit acceptance gates, and writes reports that someone else can recompute from the original records.

The demonstration replays saved Docker runs of scripted controls. No installation, account or model key is needed to explore it. Research preview · MIT · Python 3.11+ · Linux for local workflows · no third-party Python runtime dependencies.

Review a real model upgrade

Compare Qwen3-8B and Qwen3.8-27B-FP8: 8 public support-planning cases, 3 fresh generations per configuration, all 48 original answers and native pytest checks. Download the comparison or regrade it offline. Plans are not executed; model size and quantization differ. Complete plans improve 15/24 → 19/24, while 10 named checks lose passes and the upgrade gate fails. These checks share two format/schema failures. Protocol and reproduction.

Start with one review

1. See the regression. Compare the two revisions, then inspect retry-after-commit in the case explorer.
2. Check the decision. Compare the acceptance gates: the same 93.75% score passes a permissive rule and fails the strict notes rule.
3. Recompute it locally. The first-review walkthrough installs the published wheel, downloads the records and rebuilds the comparison. Verification exits 0 for consistency; comparison exits 1 for the regression.

Install the released reviewer from PyPI in a fresh virtual environment:

python3 -m venv .venv
. .venv/bin/activate
python -m pip install evalarc==0.15.0
evalarc --version

PyPI and GitHub Releases provide the same wheel and source archive. Checksum-pinned installation is also available. The offline review needs no Docker, Node, GPU or model API. Follow the download and comparison commands to produce your first HTML report without cloning the source.

Bring your own work

Trace import accepts a bounded export format, not arbitrary cloud exports. Its scored controls are synthetic; the separate MCP example records actual local delivery with no evaluator scores. No live AgentCore evaluation is claimed.

Trying your own records? Tell us where the first review helped or got stuck. A minimal redacted example is enough; a failed setup is useful feedback too.

Gate a pull request on the results you already have

evalarc diff pairs every case and check in two Inspect AI logs, promptfoo --output files or JUnit XML reports. It exits 1 when a check that passed on the baseline fails, passes less often, is skipped or disappears, and writes an offline report with copies of both inputs. In the recorded Inspect example, accuracy rises from 0.625 to 0.8125 while three checks lose passes.

- uses: noteflowai/evalarc@v0.15.0 # or a full commit SHA
  with:
    baseline: evals/baseline.json
    current: results/current.json

See the CI gate guide for format mapping, baseline patterns and pull request comments.

What the recorded evidence covers

The saved audits detect 21/21 declared faults across three task packs. Six faults depend on one detecting case each. Removing a sole detector lowers a fresh audit's mutation score; these margins expose that dependency before the change. They do not establish coverage of unseen faults. Inspect coverage · Methodology · 251 audit case records.

For deeper exploration: repeated attempts, TOML suites and CI, Python/JavaScript candidates, recorded GPU research pilots, architecture and papers.

Review independent-source task outcomes. The SWE workflow review retains 36 GPU attempts on three public source tasks across four fixed conditions. 31 have assessable native reports; five remain uncertain because of upstream infrastructure flags. None obtained acceptance. Inspect the actual MCP preloads, eight nonempty patches, tool failures and six upstream controls. Methods and offline records.

Does “finished” mean the task passed? Equal-length context controls retain twelve Qwen3-8B attempts in two separate cohorts. Relevant guidance and unrelated prose each use a 476-token MCP payload. Inspect protocol timeouts, unchanged starter programs, numerical errors and every independent case check. Each cohort resolves 0 of 6 tasks; the diagnostic follow-up is a public development experiment, not a held-out efficacy result. Methods and offline verification.

Continue from a reviewed session, then check the delivered work. Funes MCP handoff records six Qwen3-4B continuations of one public Qwen3-8B program. The memory condition retrieves through the actual MCP entrypoint; all six programs remain unchanged and score 87.5%, with no task fully resolved. Inspect the source passages, commands and independent coordinate checks. Use the source example · Methods and offline evidence.

Carry the reviewed skill into the next session. Pinned skill handoff follows an earlier MCP skill load into six new continuations. Both conditions receive the exact historical skill through workflow-selected MCP; one also offers Funes retrieval. Six preloads and six historical retrieval results succeed, but every program remains unchanged and full acceptance is 0/6. Methods and complete evidence. This uses a different prior session from the no-skill handoff above.

Does a correct answer file prove the delivered program works? Three native Harbor controls separate answer reward from independently executed code. One control receives 100% answer reward but its program scores 80% and fails strict acceptance. These are declared scripted controls, with raw ATIF and an offline evidence bundle; no model inference or unseen-exploit claim. Feature history lives in the changelog.

Run an audit

To execute the built-in Python reference and eight deliberate coding faults, install the wheel above, then use Docker:

docker pull python:3.12-slim
evalarc audit --seeds 17 41 97 --output runs/audit

Open runs/audit/index.html. Exit 0 means the reference passed and all declared faults were detected in their intended dimensions; 1 means an audit/candidate failed, and 2 means invalid input or an environment failure.

For the bundled trusted controls, this shorter CPU-only run uses the host:

evalarc audit --task support-routing --backend local --trust-local --output runs/support-audit

Local execution has your user privileges. Use Docker for candidate isolation; see execution boundaries and readiness checks. Use a fresh output path for another run. To develop EvalArc itself, see Development.

Coding task

There are 15 cases per seed. Scores average cases within a dimension, then apply the weights above. Full resolution requires every case to pass. The eight controls cover false acknowledgements, memory-only storage, partial batches, unconditional CAS, weak JSON equality, ignored deletes, invalid keys, and commits deferred until exit.

In the bundled Docker audit, the Boolean/number equality defect earns a 0.925 partial score but fails full resolution. The report identifies the specific CAS check that detects it. Partial progress and acceptance are separate.

The grader computes expectations outside the candidate container; it never accepts a candidate's claimed reward. The SQLite reference and the in-memory oracle use different implementations. Each report records the candidate, grader, and case fingerprints, runtime limits, seeds, and resolved container image ID.

Evaluate a coding agent's output

evalarc init workspace/durable-kv
# Give this workspace and its TASK.md to your coding agent.
# After it edits main.py:
evalarc evaluate workspace/durable-kv --seeds 17 41 97 --output runs/candidate

For the coding pack, the CLI evaluates completed artifacts; it does not record the process that produced them. Use evalarc init --reference workspace/reference to create the positive control. Custom entrypoints are described in the candidate command guide.

Tool-using agents

evalarc tasks
evalarc init workspace/support --task support-routing --reference
evalarc evaluate workspace/support --task support-routing --output runs/support

Replace the scripted reference with a policy that speaks the support JSONL protocol. The evaluator sends observations; the candidate requests tool operations or finishes. Only the host's resulting ticket state determines business scores. Claimed success has no scoring authority. A case is resolved only when every check passes.

In the recorded support audit, retrying a committed note with a new idempotency key earns 0.9375 but fails acceptance because it duplicates the note. The trace shows the error, retry, and state changes.

An independent JavaScript policy demonstrates a non-Python entrypoint:

evalarc evaluate examples/support-node --task support-routing \
  --backend local --trust-local --output runs/support-node

This command requires Node.js. The Python core has no third-party runtime dependencies. EvalArc does not call an LLM or provision model credentials.

Checkpoint analysis

For progress over time, save checkpoint evaluation JSON together with elapsed seconds measured by your experiment harness:

evalarc trajectory checkpoints.json --budget-seconds 3600 --output runs/trajectory.json

The checkpoint format and scoring rules include regression handling and comparability checks. Missing agent tokens and costs remain null. Caller-reported elapsed time is not a METR time horizon.

Project scope

EvalArc audits graders and reviews evaluation evidence for coding and tool-using agents. Executable tasks, outcome checks outside the candidate, revision comparisons and portable reports help developers assess whether results meet acceptance requirements and grading rules detect declared defects.

The project serves as an audit layer within existing evaluation environments and experiment workflows. It focuses on three questions: does the result meet the task contract, did a change introduce a regression, and can the conclusion be checked against the original records? See the architecture and methodology for task contracts, scoring rules and integration scope.

Native Harbor task export, oracle/NOP execution and ATIF 1.8 records are available as bounded research integrations. A general production adapter, Prime Intellect integration and calibrated long-horizon task sets remain future work.

Research basis

Software-engineering agent evaluation, executable training environments and verifier reliability inform the design. The research report explains the technical motivation and engineering references; the paper catalog records paper sources, versions and publication status as of the documented search.

Scope and evidence

The public records validate graders and the review workflow. Tasks, reference implementations, fault controls and random seeds are public. Detection results apply to the declared defects, selected cases and recorded execution conditions. Changing seeds alone does not create an independent held-out evaluation or establish that the data was excluded from training.

The coding audit, support audit and validation record provide inspectable implementation evidence. Coverage of unknown defects, resistance to reward hacking, model capability rankings and training transfer require separate evaluation with independent data and an appropriate experimental design.

Development

Clone the source for development and for the examples/ commands in this README:

git clone https://github.com/noteflowai/evalarc.git
cd evalarc
python3 -m venv .venv
. .venv/bin/activate
python -m pip install -e ".[dev]"
pytest -q
ruff check .
ruff format --check .
python -m build

See CONTRIBUTING.md, SECURITY.md, migration notes, and LICENSE. The task-author guide explains the current built-in extension points. CI includes Python checks, the Node policies, and Docker audits for all three task packs.