any-to-bench

Convert any exam materials — photos of exam papers, exam PDFs, solution PDFs/photos, official answer keys, scoring rubrics — into a machine-gradable benchmark.

Give it everything you have for one exam; it produces an exam bundle: the structured exam, a strict answer-sheet JSON Schema for any LLM harness, and a grading spec. Fixed-answer questions grade deterministically; open-ended questions are graded by multimodal LLM judges with rubrics extracted from your materials. All common paper-exam question types are supported, including nested sub-questions, figures, tables, and LaTeX math.

Install

Requires Python ≥ 3.12.

uv tool install any-to-bench   # or: pip install any-to-bench
a2b --help

Set the API keys for the providers you use as environment variables (or in a .env file in your working directory): OPENAI_API_KEY, GOOGLE_API_KEY, HF_TOKEN, ... Google models can also run through Vertex AI on a service account — google-cloud:gemini-3.7-flash with GOOGLE_APPLICATION_CREDENTIALS set; see docs/models.md.

For development, clone the repo and:

uv sync
cp .env.example .env   # fill in the API keys for the providers you use

Usage

# 1. Ingest: any mix of PDFs and photos for ONE exam -> a bundle
a2b ingest exam.pdf answer-key.jpg rubric.pdf -o out/bundle --model openai:gpt-5.6-sol

# Add one public corpus shared by every question (not sent to the ingest model)
a2b ingest questions.pdf answer-key.pdf --resources ./corpus \
    -o out/retrieval-bundle --model codex:gpt-5.6-sol

# 2. Check the bundle
a2b validate out/bundle

# 3. Have an LLM take the exam (any provider — this is the benchmark part)
a2b solve out/bundle --model google:gemini-3.7-flash -o out/answers.json

# 4. Grade the answer sheet
a2b grade out/bundle out/answers.json -o out/report.json
# override judge model(s): --judge-model openai:gpt-5.6-sol --judge-model agy:gemini-3.7-flash-high

# Or benchmark several models at once: solve + grade each, compare in one table
a2b bench out/bundle -o out/bench \
    --model openai:gpt-5.6-terra --model google:gemini-3.7-flash

# Share bundles via Hugging Face datasets (viewer-friendly, byte-faithful round trip)
a2b upload out/bundle user/my-exams --name matha
a2b download user/my-exams --name matha -o local/bundle

# Publish what you measured, so it accumulates into a leaderboard
a2b results publish out/bench user/my-results --source-repo user/my-exams

a2b is a shorthand alias for any-to-bench — every command works with both. In a cloned repo without installing, prefix commands with uv run (e.g. uv run a2b ...).

Ingest, solve, and judge models are independent. Use a codex:, claude:, or agy: model string (e.g. codex:gpt-5.6-sol, claude:opus, agy:gemini-3.7-flash-high) to run a phase agentically via that CLI instead of direct LLM calls — same commands, same outputs. All commands accept --effort and report token usage.

Existing document- or repository-based questions can be packaged as resource-backed retrieval benchmarks. Agentic takers receive the complete original corpus; direct LLMs get bounded read-only search/list/read tools over strict UTF-8 text. Actual file/byte exposure and optional, score-neutral citation checks are retained through benchmark and publication.

Example dataset

JacobLinCool/taiwan-exams is a corpus built with this tool: 182 Taiwanese national exams, 7,772 questions, ingested from the official papers, answer keys, and marking schemes with codex:gpt-5.6-sol (cap with claude:claude-opus-5). One subset per exam, named <exam>-<year>-<subject>, where the year is the ROC year the exam is named for (113–115 = 2024–2026):

英語（聽力）is the one paper left out: its questions are spoken and published as audio, and a bundle carries text and images.

The dataset viewer shows the extracted questions with their figures embedded; <subset>/bundle/ holds the bundle itself. Benchmark against any of them:

a2b download JacobLinCool/taiwan-exams --name gsat-115-math-a -o bundle
a2b solve bundle --model google:gemini-3.7-flash -o answers.json
a2b grade bundle answers.json -o report.json

Published results

JacobLinCool/taiwan-exams-results holds the first scores against that corpus: nineteen taker configurations over all 21 papers of the 115 year, 1,748 points apiece — four agentic models (codex:gpt-5.6-sol, codex:gpt-5.6-luna, claude:claude-opus-5, claude:claude-sonnet-5) at each of low, medium, high and xhigh, plus google-cloud:gemini-3.7-flash through Vertex AI at low, medium and high. Sixteen of them also sat the 40 papers of the 115 統測 — all four agentic models at all four efforts — so those sixteen rows cover 61 papers and 5,748 points.

Over the 21 papers every configuration sat:

Eight of the nineteen; the dataset card ranks all of them. Effort is not the whole story, and neither is spending: four rows are all that survive on the cost/score frontier, and codex:gpt-5.6-sol holds three of them. Its dial barely moves the rule-graded column — 99.2%, 99.0%, 99.2% at medium, high and xhigh — so what separates its top three rows is the judged half, and that half is only as good as the judge model named in the entry.

On the 40 papers of the 統測 — 4,000 points, 1,844 questions, only 9 of them judged — codex:gpt-5.6-luna scores 84.8%, 91.9%, 96.4% and 98.2% at low, medium, high and xhigh. The dial is flatter there than on 會考/學測/分科 (13.4 points of range against 19.3), because 37 of the 40 papers are multiple choice throughout and a low-effort reader already does well on them. The dials differ by an order of magnitude in slope: luna covers 13.4 points, claude: claude-sonnet-5 2.1 (93.8% → 95.9%), claude:claude-opus-5 0.9 (97.3% → 98.2%) — opus starts where luna finishes. codex:gpt-5.6-sol is done before its dial begins: 97.1% at low, 99.9% at medium, and 4000/4000 at xhigh — every one of the 1,835 rule-graded questions and all 80 judged points, on 393k output tokens, less than luna's xhigh spends to finish 1.8 points behind. Every point on the cost/score frontier but the cheapest is a sol row; the other eleven configurations all spend more for less.

google-cloud:gemini-3.7-flash is the first taker here that is a plain API model rather than a coding agent, and the difference shows in a way worth naming: it loses 41, 34 and 24 points at low, medium and high to answering fill-in-blank questions with its own blank ids instead of the schema's, in the maths papers, even after the harness retries with the error. Following the answer schema is part of sitting the exam, so those points stay lost — but the gap is a formatting failure, not arithmetic, and the entries say so.

Rule-graded points are scored by program, so that column compares across any two rows; judged points depend on the judge model, which is named per entry. One run per paper, so there is no error bar — read small gaps as unresolved.

a2b bench bundle -o out --model your:model --effort high
a2b results publish out user/your-results --source-repo JacobLinCool/taiwan-exams

Browse it as a leaderboard — pick the papers, decide whether judged questions count, and compare cost against score. See docs/results.md for the layout.

Design principles

The three phases have deliberately asymmetric goals:

• Ingest: spend freely, be exact. A bundle is a dataset — built once, reused by everyone who ever benchmarks against it. Extraction accuracy is worth almost any model cost and wall time; this is why ingestion supports the expensive agentic mode, gap-repair rounds, and validate-and-fix loops. Intelligence spent here is amortized across every future run.
• Solve: no constraints. The taker is the thing being measured — anything from a cheap LLM call to a full agent belongs here.
• Grade: require as little intelligence as possible. The same answer sheet must earn the same score every time. Fixed-answer questions grade as pure scripts — zero model calls, bit-for-bit reproducible. Where an LLM judge is unavoidable (open-ended questions), it is constrained, not creative: precise rubrics with defined point levels, reference answers, and level snapping mean the judge follows the rubric mechanically instead of improvising — so even a non-frontier judge model grades accurately and consistently.

Put differently: ingest converts intelligence into structure (keys, rubrics, schemas) exactly once, so that grading needs almost none, forever.

Documentation

• The exam bundle — output format, question model, validation
• How ingestion works — the LLM-mode extraction pipeline
• Agentic mode — CLI backends, workspaces, and the fix loop
• Grading semantics — deterministic rules and LLM judges
• Benchmarking — the bench model matrix and its metrics
• Publishing — sharing bundles as Hugging Face datasets
• Publishing results — leaderboard entries, and how scores are counted
• Models, effort, usage — model strings, --effort, token reporting

Development

uv run pytest -q        # fully offline — model requests are forbidden in tests
uv run ruff check .
uv run ruff format .    # CI enforces this with --check

The test suite fakes the LLM layer (any_to_bench.llm.build_agent) and the agentic subprocess layer (run_codex / run_claude / run_agy), so the entire ingest → solve → grade pipeline runs end-to-end in every mode without network access or any CLI binary installed.