Prebuttal

Answer Reviewer 2 before Reviewer 2 exists.

Purpose-built for ML conferences: real NeurIPS / ICLR / ICML / ARR review-form output, three reviewers running in fully independent contexts (optionally cross-model), and a built-in pipeline to measure its scores against real OpenReview ratings.

Prebuttal is a pre-submission stress test for your paper: it sends your submission through three isolated AI reviewers with different temperaments plus an area chair, and hands you a conference-format review report — scores, disagreement, weaknesses with citations to your own tables — before the real Reviewers 1–3 see it. See a synthetic sample: report · share page — or what a deliberately flawed (fictional) paper gets: 1/10 report.

Why not just paste the paper into ChatGPT?

That is the main competitor. It loses, four ways:

Quickstart

pip install prebuttal

prebuttal run paper.pdf --venue iclr
prebuttal run https://arxiv.org/abs/2510.12345 --venue neurips
prebuttal run ./latex-src/ --venue acl

Prefer an isolated install for command-line tools? pipx install prebuttal or uv tool install prebuttal work the same way (or uvx prebuttal ... to run without installing). Requires Python 3.11+.

The first run costs zero setup if you already have a subscription CLI or Ollama — Prebuttal auto-detects the cheapest path at startup and prints what it chose (prebuttal backends to inspect):

Unpublished manuscript? The text is sent to the backend's provider, and what happens to it there is that provider's privacy policy, not ours — see Privacy & data custody. For local-only review use ollama or relay.

⚠️ Free tiers are usually paid for with your data. Free API tiers (the Gemini free tier, OpenRouter :free models and their upstream providers) may let the provider log, review or train on what you send. They are fine for published papers and trying the tool out — for an unpublished manuscript, check the provider's current terms, use a paid tier, or stay local.

Useful flags:

prebuttal run paper.pdf --venue neurips --dry-run   # cost + plan preview, calls nothing
prebuttal run paper.pdf --venue iclr --json out.json  # also write machine-readable JSON
prebuttal run paper.pdf --note "added ablation"     # label this revision
prebuttal run paper.pdf --seed 42                   # reproducible (API / Ollama backends)
prebuttal run paper.pdf -y                          # skip the plan confirmation

In auto mode on a terminal, Prebuttal shows the review plan before calling anything — press Enter to run, e to re-pick each reviewer's persona, backend and model via numbered menus, or n to abort. CI and pipes get the plan printed and proceed automatically; fully-explicit configs never prompt.

prebuttal models lists every backend's actually-available models (live API scan + installed Ollama models + claude aliases) with the current default marked — pick one and set model = "..." per reviewer in prebuttal.toml. Built-in defaults are mid-tier models current at release time and change between versions — providers retire model ids quickly, so if a default stops working, upgrade prebuttal or pin a model yourself.

The reviewers

Every persona enforces the same protocol: cite the evidence (section, table, number) or the point doesn't count; use the full scale; JSON-only output. There is also an easter-egg roast persona (persona = "paper_roast") for when you need the truth fast.

Personas live in prebuttal/personas/ and venue forms in prebuttal/venues/forms/ — plain text files, PRs welcome (scale wording drifts every year; keep it current).

Your own venue (custom review forms)

Built in: neurips, iclr, icml, acl. Anything else — CVPR, AAAI, a journal, this year's revised ICML form, your lab's internal mock review — is one TOML file plus the form text:

prebuttal venues new cvpr              # template in ~/.prebuttal/venues/
prebuttal venues new icml-2027 --from icml --here   # copy a built-in, project-local
prebuttal venues                       # list built-in + custom (invalid ones flagged)
prebuttal venues show cvpr             # the scores and form reviewers will see
prebuttal run paper.pdf --venue cvpr   # or --venue path/to/venue.toml

# cvpr.toml
name = "CVPR 2027"
form = "cvpr.md"          # the reviewer guidelines, shown to reviewers verbatim
rating = "rating"         # which score is the overall recommendation

[[fields]]
key = "weaknesses"
label = "Weaknesses"
kind = "list"             # text | list | score

[[fields]]
key = "rating"
label = "Overall recommendation"
kind = "score"
max = 6                   # scores run 1..max; half_steps = true allows 3.5
anchors = { "1" = "Reject", "3" = "Borderline", "6" = "Strong accept" }

Venues are looked up in ./.prebuttal/venues/ (commit it to share with co-authors) and then ~/.prebuttal/venues/; built-in names can't be shadowed, so a copy needs its own name. Files are checked before any model is called, and every error names the file and the setting. Reports keep the venue's definition, so archived runs still render after the file changes. Custom venues are not calibrated: the calibration script needs OpenReview scores for a real venue. A venue's form text is instructions to your reviewers, so only use venue files you trust (the form must sit next to its .toml).

Configuration

prebuttal init writes a commented prebuttal.toml. Per-reviewer backends, models and temperatures:

[default]
venue = "iclr"
seed = 42
# effort = "high"             # optional, see "Reasoning effort" below

[[reviewer]]
id = "R1"
persona = "senior_kind"
backend = "agent-claude"      # claude -p, subscription auth
temperature = 0.4

[[reviewer]]
id = "R2"
persona = "adversarial"
backend = "agent-codex"       # codex exec, subscription auth
model = "gpt-5.1"             # optional: forwarded as --model (agent CLIs too)
temperature = 0.7

[[reviewer]]
id = "R3"
persona = "detail_obsessive"
backend = "api-gemini"        # free tier; default model gemini-3.5-flash-lite
temperature = 0.5

[ac]
persona = "area_chair"
backend = "agent-claude"

Reasoning effort

Leave it unset and every model thinks as much as its provider's default — the right choice for most papers. To trade cost and time for depth, set --effort low|medium|high for one run, effort = "..." under [default], or per [[reviewer]] / [ac]. It maps onto each backend's own control:

Where a model has no such control (classic chat models, relay, custom agent CLIs) the setting is skipped; --dry-run says so up front, and every report lists the effort each review actually ran with.

Figures as images

Reviewers see figure captions by default. With --figures (or figures = true under [default]), reviewers on the vision-capable API backends — api-anthropic, api-openai, api-gemini, api-openrouter — also get the figures as images: for a PDF or arXiv paper, each page that holds a figure caption; for a LaTeX project, the first \includegraphics of each figure (files inside the project folder only). At most max_figures (default 6) images, ~1.5k tokens each. A text-only model behind one of those APIs is retried without images; agent CLIs, Ollama and relay keep captions only. Each review says how many images it saw.

Calibration (honesty, not prophecy)

Scores are meaningless unless measured. The calibration pipeline runs Prebuttal on papers that already have real OpenReview reviews and plots predicted vs true mean ratings:

The script lives in the repository (it is not part of the pip package), so run it from a clone:

git clone https://github.com/GuoYuKai-SaMuEl/prebuttal && cd prebuttal
pip install -e ".[calibration]"
python scripts/calibration/calibrate.py \
    --venue-id ICLR.cc/2024/Conference --venue iclr -n 20 --out-dir calib_out

Output: scatter plot + MAE + correlation. No calibration results are published yet — until they are, treat every score as uncalibrated. Once published, the claim they support is narrow and deliberate: this tool was measured — unlike role-play prompts. It is not a claim that it predicts your score; real reviewers disagree with each other by more than this tool's error. Disagreement (σ) is reported with every aggregate for exactly that reason.

The revision plan

Reviews tell you what is wrong; the revision plan tells you what to do about it. The area chair merges what the reviewers raised into a short, prioritised list — no extra model call, it comes with the meta-review:

- [ ] **must** — Baseline set is thin: Table 1 misses a Perceiver comparison
      → Add Perceiver-AR to Table 1 or scope the claim to the baselines run (R2, R3 · effort: medium)
- [ ] **should** — No variance across seeds for the Table 2 ablation
      → Report mean and std over 3 seeds (R3 · effort: small)

It opens every report, the run prints the top items (Fix first:), and prebuttal plan [run] prints it as a checklist to paste into your notes or an issue (--json for scripts). After you revise and re-run, prebuttal diff checks the old plan against the new reviews: which items are no longer detected, which still are, and what is new. The matching is by wording, so a reworded issue shows up as "not re-detected" and "new" — compare runs made with the same backends and seed, and read both lists before ticking anything off.

Projects, reports & revision history

Each paper gets a project, found from where it came from: the same PDF path, LaTeX folder or arXiv id always lands in the same project, even when the title changes between revisions (--project NAME to pick one yourself). Every run gets a number within it — mamba-linear-time-sequence#3 — and its own report folder, so nothing is ever overwritten:

prebuttal-reports/                    (has a .gitignore: reviews stay out of git)
└── mamba-linear-time-sequence/
    ├── latest.md / latest.html       ← always the newest run
    ├── 2026-10-25_1530_iclr/report.md, report.html
    └── 2026-10-26_0910_iclr/report.md, report.html

The complete record (Markdown, HTML, JSON and the resolved config — backends, models, seeds) is also archived under ~/.prebuttal/ (--no-save opts out), so everything below works from any directory:

prebuttal projects                   # one line per paper: runs, latest score, trend
prebuttal runs [project]             # its runs, newest first, with notes
prebuttal open                       # newest report in the browser (HTML)
prebuttal plan                       # the newest run's revision plan as a checklist
prebuttal diff                       # newest run vs the one before: ratings, weaknesses, plan
prebuttal diff --llm                 # ... matching plan items by meaning (one model call)
prebuttal retry                      # re-run only the reviewers that failed (+ the AC)
prebuttal dashboard                  # local HTML overview: every project, score trends, links
prebuttal show / export / note / rm  # print, copy out, annotate, delete runs
prebuttal projects rename|rm         # tidy up

Runs are referenced as latest (the default), mamba (newest run of a project), mamba#3 (run 3), mamba~1 (one before the newest) or by id prefix.

The revision loop: fix what the reviewers flagged, re-run with the same --seed and a --note "what changed", then prebuttal diff shows which weaknesses went away, persisted, or appeared, and which revision-plan items are no longer detected. Report location and formats are configurable: --out-dir, --format md,html,json, or an [output] section in prebuttal.toml (dir = "" keeps reports in the archive only). Need a file at a fixed path, e.g. for a script? --json PATH, or prebuttal export -f md -o PATH after the run.

Skill package

skills/prebuttal/SKILL.md teaches any agent to drive the CLI and interpret the JSON — including relay mode (the reviewers are the agent's own subagents) and the runs/diff revision loop. The skill file is explicit about the one rule that matters: the agent must run the CLI, never role-play the reviewers itself.

When the CLI detects it is running inside a coding agent (currently Claude Code and Gemini CLI), prebuttal run and prebuttal backends print a one-line hint pointing at relay mode. An agent environment that should always use its own subagents can pin it once with PREBUTTAL_BACKEND=relay. Relay is never chosen automatically: the CLI has to run in the background while the agent answers the requests, or both sides wait on each other.

Install it into your agent's skills directory — for Claude Code:

mkdir -p ~/.claude/skills/prebuttal && curl -fsSL -o ~/.claude/skills/prebuttal/SKILL.md \
  https://raw.githubusercontent.com/GuoYuKai-SaMuEl/prebuttal/main/skills/prebuttal/SKILL.md
# or, from a clone: cp -r skills/prebuttal ~/.claude/skills/

Other agents (ZCode, Codex, …): copy or symlink skills/prebuttal/ wherever your agent discovers skills (see its docs), or reference the file from your AGENTS.md / system prompt.

Reference

Environment variables

API keys — two ways, pick either:

prebuttal key set gemini     # quick setup: hidden prompt, stored for next time
# or the classic way: set GEMINI_API_KEY / OPENAI_API_KEY / ANTHROPIC_API_KEY yourself

Stored keys live in ~/.prebuttal/keys.env (user-only permissions, like ~/.aws/credentials); real environment variables always take precedence, so CI setups are unaffected. prebuttal key status shows what's set (masked), prebuttal key unset <name> removes one.

API keys are only read from the environment or keys.env — never written to reports, JSON, or share pages.

Exit codes

Comparison

What Prebuttal provides, next to the common alternatives (by category — check any specific tool's own docs for what it supports):

How it works

PDF / LaTeX dir / arXiv link
   └─ ingestion: sections, cell-accurate tables, figure captions, refs
        └─ reviewer pack (budget-aware; tables are never truncated)
             ├─ R1 senior_kind ──┐ three independent contexts,
             ├─ R2 adversarial ──┤ parallel fan-out, venue form + JSON schema
             ├─ R3 detail_obsessive ─┤
             └─ AC area chair ◄──┘ meta-review + estimated rating + disagreement
                  └─ Markdown + HTML report per run · JSON · project history

Orchestration lives in the CLI, not the host agent — the same pipeline runs from a terminal, from inside any coding agent, or in CI.

Roadmap & versioning

Versioning policy while pre-1.0: patch releases (0.1.1, 0.1.2, …) are fixes and small additions; minor releases (0.2, 0.3, …) are substantial new features; 1.0 only once the interface is battle-tested.

Tentative — priorities shift with usage and feedback. Short-term the focus is hardening the core on real papers, not adding features:

1. v0.2 (released) — core: ingestion pipeline, real venue forms, 3 independent reviewers + AC, every backend (agent CLIs / APIs / Ollama / relay), reports + JSON + share page, calibration script; plus projects & revision history (per-paper report folders, run management, local dashboard) and per-role reasoning effort; v0.2.1 hardened PDF table extraction on real two-column papers
2. v0.3 (current) — custom venues (any review form and scale as a TOML file) and the revision plan (the AC's prioritised fix list, checked against the next run by prebuttal diff)
3. Now (v0.3.x) — polish from real-usage findings: calibration scatter plot + honest per-model calibration notes, provider-reliability UX (free-pool reality, retry transparency), relay UX
4. Next (v0.4+) — community personas, OpenReview submission integration, and a browser-only web quick-scan (GitHub Pages: drag-drop a PDF + your own OpenRouter/Gemini key — no server, we stay a non-custodian); then the rebuttal workbench (reviews → point-by-point response letter) and "CI for papers" (GitHub Action: \ref/\label checks, citation sanity)
5. Eventually — the calibration script becomes a public benchmark / leaderboard: tools depreciate, benchmarks appreciate.

Develop

git clone https://github.com/GuoYuKai-SaMuEl/prebuttal && cd prebuttal
python -m venv .venv && .venv/Scripts/pip install -e ".[dev]"   # Windows
.venv/bin/pip install -e ".[dev]"                               # macOS/Linux
pytest -q

Contributions welcome — the easiest ones are venue forms, personas, and backends; see CONTRIBUTING.md for the three-step guide. Also read CHANGELOG.md, SECURITY.md, and our Code of Conduct.

Honesty & policy

• Prebuttal labels every review with the backend and model that produced it.
• It reports score disagreement, always. High σ is information.
• It is a stress test, not a predictor — the report footer says so, every time.
• Mind conference policies: with NeurIPS 2026 running an official AI-assisted reviewing experiment, the etiquette is clear — AI to prepare your own paper is yours to decide; AI to review others' papers must follow each venue's disclosure rules.

Privacy & data custody

• Your paper's text goes to whatever backend you configure. With API or agent-CLI backends the manuscript is transmitted to that provider — and OpenRouter further routes each request to a third-party upstream provider. What happens to the text there (retention, training use, human review) is governed solely by each provider's terms and privacy policy, not by Prebuttal.
• Prebuttal is an integration layer, not a data custodian. This project operates no models and runs no servers; we never see your paper. For unpublished results, check the backend provider's data-use terms before sending — provider-side handling of your data is the provider's responsibility, not this project's.
• Free tiers deserve extra care: free API tiers (e.g. Gemini's free tier, OpenRouter :free models) may allow logging or training on inputs; paid tiers usually don't. Read the terms before sending unpublished work.
• Local-only is a first-class path: ollama never sends text off your machine (as long as PREBUTTAL_OLLAMA_HOST / OLLAMA_HOST point at localhost — a remote Ollama server receives the paper like any API), and relay leaves who answers entirely to you.
• What Prebuttal itself stores stays on your disk: run history (pack text, raw outputs, resolved config) under ~/.prebuttal/runs/ and keys in ~/.prebuttal/keys.env (user-only). No telemetry. Note reports and share pages contain excerpts of your paper — sharing them publishes those excerpts.

License & trademarks

Prebuttal is an independent project, not affiliated with or endorsed by NeurIPS, ICLR, ICML, ACL/ARR or any model provider. Venue names are used only to describe which review form a report follows; the forms are paraphrased summaries of the public reviewer guidelines.

MIT