Pergamon Bench — eval

The code behind Pergamon Bench, a benchmark of legal reasoning and research tasks on continental-European law, run with Harbor.

To run the benchmark, start with the Pergamon Bench README. It covers installing, running, fitting the run to your machine, results and pergamon-bench validate.

This repository holds:

• pergamon-bench (src/pergamon_bench/): the agent wrapper Harbor loads (OpenCode with the official settings), the job file pergamon-bench init writes, and the pergamon-bench command;
• pbench-grader (packages/pbench-grader/): the grading;
• pbench-search (packages/pbench-search/): the agent's web search tool.

Documentation for contributors and the curious, in docs/:

• architecture.md: how the pieces fit together, and what "official" means in code;
• grading.md: how answers are graded, and every grading setting;
• grading-versions.md: what each grading version pins, and why it changed;
• methodology.md: the grader card;
• failures-and-retries.md: which failures are retried instead of graded;
• web-search.md: the web tools and search providers;
• images.md: the sandbox and grading images;
• private-set-process.md: evaluating on the private set;
• development.md: tests, lint and releases.

CHANGELOG.md lists every release; changes that can move scores are marked [score-impact].

Releasing to PyPI

The pergamon-bench package (one wheel: pergamon_bench and pbench_grader) is published by the release workflow on every v* tag, through PyPI trusted publishing (no token). Bump version in pyproject.toml and packages/pbench-grader/pyproject.toml to the same value, update CHANGELOG.md, push, then:

gh release create v0.8.0 --title "v0.8.0" --notes "See CHANGELOG.md"
gh run watch

The workflow refuses a tag that does not match pyproject.toml's version. The images (workspace-v*, verifier-v* tags) are released separately: see images.md.