Prebuttal

Answer Reviewer 2 before Reviewer 2 exists.

Purpose-built for ML conferences: real NeurIPS / ICLR / ICML / ARR review-form output, three reviewers running in fully independent contexts (optionally cross-model), and a built-in pipeline to measure its scores against real OpenReview ratings.

Prebuttal is a pre-submission stress test for your paper: it sends your submission through three isolated AI reviewers with different temperaments plus an area chair, and hands you a conference-format review report — scores, disagreement, weaknesses with citations to your own tables — before the real Reviewers 1–3 see it. See a synthetic sample: report · share page — or what a deliberately flawed (fictional) paper gets: 1/10 report.

Why not just paste the paper into ChatGPT?

That is the main competitor. It loses, four ways:

Quickstart

pip install prebuttal

prebuttal run paper.pdf --venue iclr
prebuttal run https://arxiv.org/abs/2510.12345 --venue neurips
prebuttal run ./latex-src/ --venue acl

Prefer an isolated install for command-line tools? pipx install prebuttal or uv tool install prebuttal work the same way (or uvx prebuttal ... to run without installing). Requires Python 3.11+.

The first run costs zero setup if you already have a subscription CLI or Ollama — Prebuttal auto-detects the cheapest path at startup and prints what it chose (prebuttal backends to inspect):

Unpublished manuscript? The text is sent to the backend's provider, and what happens to it there is that provider's privacy policy, not ours — see Privacy & data custody. For local-only review use ollama or relay.

⚠️ Free tiers are usually paid for with your data. Free API tiers (the Gemini free tier, OpenRouter :free models and their upstream providers) may let the provider log, review or train on what you send. They are fine for published papers and trying the tool out — for an unpublished manuscript, check the provider's current terms, use a paid tier, or stay local.

Useful flags:

prebuttal run paper.pdf --venue neurips --dry-run   # cost + plan preview, calls nothing
prebuttal run paper.pdf --venue iclr --json out.json --share   # machine output + share page
prebuttal run paper.pdf --seed 42                   # reproducible
prebuttal run paper.pdf -y                          # skip the plan confirmation

In auto mode on a terminal, Prebuttal shows the review plan before calling anything — press Enter to run, e to re-pick each reviewer's persona, backend and model via numbered menus, or n to abort. CI and pipes get the plan printed and proceed automatically; fully-explicit configs never prompt.

prebuttal models lists every backend's actually-available models (live API scan + installed Ollama models + claude aliases) with the current default marked — pick one and set model = "..." per reviewer in prebuttal.toml. Built-in defaults are mid-tier models current at release time and change between versions — providers retire model ids quickly, so if a default stops working, upgrade prebuttal or pin a model yourself.

The reviewers

Every persona enforces the same protocol: cite the evidence (section, table, number) or the point doesn't count; use the full scale; JSON-only output. There is also an easter-egg roast persona (persona = "paper_roast") for when you need the truth fast.

Personas live in prebuttal/personas/ and venue forms in prebuttal/venues/forms/ — plain text files, PRs welcome (scale wording drifts every year; keep it current).

Configuration

prebuttal init writes a commented prebuttal.toml. Per-reviewer backends, models and temperatures:

[default]
venue = "iclr"
seed = 42

[[reviewer]]
id = "R1"
persona = "senior_kind"
backend = "agent-claude"      # claude -p, subscription auth
temperature = 0.4

[[reviewer]]
id = "R2"
persona = "adversarial"
backend = "agent-codex"       # codex exec, subscription auth
model = "gpt-5.1"             # optional: forwarded as --model (agent CLIs too)
temperature = 0.7

[[reviewer]]
id = "R3"
persona = "detail_obsessive"
backend = "api-gemini"        # free tier; default model gemini-3.5-flash-lite
temperature = 0.5

[ac]
persona = "area_chair"
backend = "agent-claude"

Calibration (honesty, not prophecy)

Scores are meaningless unless measured. The calibration pipeline runs Prebuttal on papers that already have real OpenReview reviews and plots predicted vs true mean ratings:

The script lives in the repository (it is not part of the pip package), so run it from a clone:

git clone https://github.com/GuoYuKai-SaMuEl/prebuttal && cd prebuttal
pip install -e ".[calibration]"
python scripts/calibration/calibrate.py \
    --venue-id ICLR.cc/2024/Conference --venue iclr -n 20 --out-dir calib_out

Output: scatter plot + MAE + correlation. No calibration results are published yet — until they are, treat every score as uncalibrated. Once published, the claim they support is narrow and deliberate: this tool was measured — unlike role-play prompts. It is not a claim that it predicts your score; real reviewers disagree with each other by more than this tool's error. Disagreement (σ) is reported with every aggregate for exactly that reason.

Run history & revision diffing

Every run is archived automatically (best-effort, never blocks a review) under ~/.prebuttal/ — reports, JSON, and the resolved config (backends, models, seeds) for exact reproduction:

prebuttal runs                       # list archived runs
prebuttal show <run-id-prefix>       # re-read a report (--json for the data)
prebuttal diff <old> <new>           # ratings A→B + weakness movement

diff is the revision loop: fix what the reviewers flagged, re-run with the same --seed, and see which weaknesses went away, persisted, or appeared. Basic usage is unchanged — archiving is silent, --no-save opts out.

Skill package

skills/prebuttal/SKILL.md teaches any agent to drive the CLI and interpret the JSON — including relay mode (the reviewers are the agent's own subagents) and the runs/diff revision loop. The skill file is explicit about the one rule that matters: the agent must run the CLI, never role-play the reviewers itself.

When the CLI detects it is running inside a coding agent (currently Claude Code and Gemini CLI), prebuttal run and prebuttal backends print a one-line hint pointing at relay mode. An agent environment that should always use its own subagents can pin it once with PREBUTTAL_BACKEND=relay. Relay is never chosen automatically: the CLI has to run in the background while the agent answers the requests, or both sides wait on each other.

Install it into your agent's skills directory — for Claude Code:

mkdir -p ~/.claude/skills/prebuttal && curl -fsSL -o ~/.claude/skills/prebuttal/SKILL.md \
  https://raw.githubusercontent.com/GuoYuKai-SaMuEl/prebuttal/main/skills/prebuttal/SKILL.md
# or, from a clone: cp -r skills/prebuttal ~/.claude/skills/

Other agents (ZCode, Codex, …): copy or symlink skills/prebuttal/ wherever your agent discovers skills (see its docs), or reference the file from your AGENTS.md / system prompt.

Reference

Environment variables

API keys — two ways, pick either:

prebuttal key set gemini     # quick setup: hidden prompt, stored for next time
# or the classic way: set GEMINI_API_KEY / OPENAI_API_KEY / ANTHROPIC_API_KEY yourself

Stored keys live in ~/.prebuttal/keys.env (user-only permissions, like ~/.aws/credentials); real environment variables always take precedence, so CI setups are unaffected. prebuttal key status shows what's set (masked), prebuttal key unset <name> removes one.

API keys are only read from the environment or keys.env — never written to reports, JSON, or share pages.

Exit codes

Comparison

What Prebuttal provides, next to the common alternatives (by category — check any specific tool's own docs for what it supports):

How it works

PDF / LaTeX dir / arXiv link
   └─ ingestion: sections, cell-accurate tables, figure captions, refs
        └─ reviewer pack (budget-aware; tables are never truncated)
             ├─ R1 senior_kind ──┐ three independent contexts,
             ├─ R2 adversarial ──┤ parallel fan-out, venue form + JSON schema
             ├─ R3 detail_obsessive ─┤
             └─ AC area chair ◄──┘ meta-review + estimated rating + disagreement
                  └─ Markdown report · --json · --share page

Orchestration lives in the CLI, not the host agent — the same pipeline runs from a terminal, from inside any coding agent, or in CI.

Roadmap & versioning

Versioning policy while pre-1.0: patch releases (0.1.1, 0.1.2, …) are fixes and small additions; minor releases (0.2, 0.3, …) are substantial new features; 1.0 only once the interface is battle-tested.

Tentative — priorities shift with usage and feedback. Short-term the focus is hardening the core on real papers, not adding features:

1. v0.1 (current, pre-release) — core: ingestion pipeline, real venue forms, 3 independent reviewers + AC, every backend (agent CLIs / APIs / Ollama / relay), reports + JSON + share page, calibration script
2. Now (v0.1.x) — polish from real-usage findings: table & ingestion edge cases (complex PDF tables), calibration scatter plot + honest per-model calibration notes, provider-reliability UX (free-pool reality, retry transparency), relay UX
3. Next (v0.2+) — more venues, community personas, OpenReview submission integration, and a browser-only web quick-scan (GitHub Pages: drag-drop a PDF + your own OpenRouter/Gemini key — no server, we stay a non-custodian); then the rebuttal workbench (reviews → point-by-point response letter) and "CI for papers" (GitHub Action: \ref/\label checks, citation sanity)
4. Eventually — the calibration script becomes a public benchmark / leaderboard: tools depreciate, benchmarks appreciate.

Develop

git clone https://github.com/GuoYuKai-SaMuEl/prebuttal && cd prebuttal
python -m venv .venv && .venv/Scripts/pip install -e ".[dev]"   # Windows
.venv/bin/pip install -e ".[dev]"                               # macOS/Linux
pytest -q

Contributions welcome — the easiest ones are venue forms, personas, and backends; see CONTRIBUTING.md for the three-step guide. Also read CHANGELOG.md, SECURITY.md, and our Code of Conduct.

Honesty & policy

• Prebuttal labels every review with the backend and model that produced it.
• It reports score disagreement, always. High σ is information.
• It is a stress test, not a predictor — the report footer says so, every time.
• Mind conference policies: with NeurIPS 2026 running an official AI-assisted reviewing experiment, the etiquette is clear — AI to prepare your own paper is yours to decide; AI to review others' papers must follow each venue's disclosure rules.

Privacy & data custody

• Your paper's text goes to whatever backend you configure. With API or agent-CLI backends the manuscript is transmitted to that provider — and OpenRouter further routes each request to a third-party upstream provider. What happens to the text there (retention, training use, human review) is governed solely by each provider's terms and privacy policy, not by Prebuttal.
• Prebuttal is an integration layer, not a data custodian. This project operates no models and runs no servers; we never see your paper. For unpublished results, check the backend provider's data-use terms before sending — provider-side handling of your data is the provider's responsibility, not this project's.
• Free tiers deserve extra care: free API tiers (e.g. Gemini's free tier, OpenRouter :free models) may allow logging or training on inputs; paid tiers usually don't. Read the terms before sending unpublished work.
• Local-only is a first-class path: ollama never sends text off your machine (as long as PREBUTTAL_OLLAMA_HOST / OLLAMA_HOST point at localhost — a remote Ollama server receives the paper like any API), and relay leaves who answers entirely to you.
• What Prebuttal itself stores stays on your disk: run history (pack text, raw outputs, resolved config) under ~/.prebuttal/runs/ and keys in ~/.prebuttal/keys.env (user-only). No telemetry. Note reports and share pages contain excerpts of your paper — sharing them publishes those excerpts.

License & trademarks

Prebuttal is an independent project, not affiliated with or endorsed by NeurIPS, ICLR, ICML, ACL/ARR or any model provider. Venue names are used only to describe which review form a report follows; the forms are paraphrased summaries of the public reviewer guidelines.

MIT