refereed 0.1.0

Run a panel of seven independent AI referees over a research paper, then synthesise their reports into one recommendation.

refereed

Run a panel of seven independent AI referees over a research paper, then synthesise their reports into one recommendation.

pip install "refereed[anthropic]"
refereed review paper.md

My paper
==========================================================
  ✓  Correctness reviewer             7/10  minor_revisions
        major: Lemma 2 assumes independence that Section 3 does not establish
  ✦  Novelty reviewer                 6/10  major_revisions
  ⚙  Methodology reviewer             7/10  minor_revisions
  ❖  Citations reviewer               8/10  accept
  ↻  Reproducibility reviewer         5/10  major_revisions
        major: No code or data released; results cannot be checked
  ✎  Writing & clarity reviewer       8/10  accept
  ⚔  Adversarial (Reviewer 2)         5/10  major_revisions

  weighted score: 6.53/10
  recommendation: major_revisions

  Revisions requested:
    1. State the independence assumption in Lemma 2 explicitly.
    2. Release the experiment code, or specify conditions precisely enough to rerun.

Why seven reviewers

A single "review this paper" prompt produces a diffuse answer that hedges across every dimension at once. Seven narrow reviewers each have to commit to a position, and the disagreements between them carry information a single pass throws away. The synthesiser reads all seven and reports where they converged and where they split.

Reviewer	Remit	Weight
Correctness	Claims, internal logic, the dependency graph of the argument	1.0
Novelty	What is genuinely new against the surrounding literature	0.9
Methodology	Baselines, controls, ablations, definitions, honest limitations	1.0
Citations	Whether the bibliography does real work; pairs with citeverify	0.7
Reproducibility	What an independent researcher would need to verify the claims	0.9
Writing	Structure, hedging balance, jargon, clarity for the intended reader	0.6
Adversarial	The strongest hostile reading, stated concretely	1.0

Weights reflect what is fixable. Correctness, methodology and the adversarial reading bear on whether the work is right. Prose can be rewritten, so writing carries least.

The weighted score is computed locally from the reviewer scores. The synthesiser does not supply it. It is reproducible from the JSON output and cannot drift with the model's mood. The synthesiser's own score is kept alongside it.

Usage

CLI

refereed review paper.md                          # full panel
refereed review paper.md --json review.json       # machine-readable output
refereed review paper.md --only correctness,adversarial
refereed review paper.md --effort medium          # cheaper bulk pass
refereed review paper.md --context cites.json     # feed in citeverify output
refereed review paper.md --strict minor_revisions # exit 1 if below that bar
refereed reviewers                                # list the panel, no API call

Python

import asyncio
from refereed import review_paper, anthropic_complete

review = asyncio.run(review_paper(
    open("paper.md").read(),
    anthropic_complete(effort="high"),
    title="My paper",
))

print(review.weighted_score)
for r in review.reports:
    if r.ok:
        print(r.name, r.data["score"], r.data["recommendation"])

Failure behaviour

A panel is useful even when part of it fails, so failures are recorded rather than raised:

• A reviewer that errors or returns unparseable output is marked failed; the other six still report, and the failure is visible in the output.
• The weighted score is computed from the reviewers that succeeded.
• A failed synthesis leaves the seven reports intact.

The raw text of an unparseable response is kept so you can see what happened.

Bring your own backend

Model access goes through a single injectable callable. Nothing else in the package touches the network:

async def my_complete(system: str, user: str) -> str:
    return await my_gateway.complete(system=system, user=user)

review = await review_paper(paper, my_complete)

Point it at an internal gateway, a different provider, or a recorded fixture. The entire test suite runs through this seam, so it is a supported path.

Model

Defaults to Claude Opus 4.8 with adaptive thinking, so the model decides how much to reason per reviewer instead of working to a fixed token budget. --effort controls depth and therefore most of the cost: high by default because reviewing is intelligence-sensitive, medium for cheap bulk passes.

The core package is stdlib-only. The Anthropic SDK is an extra, needed only for live reviews, so the panel definitions, prompt construction and parsing can be imported and tested without it.

Limits, and what this is not

This is a reading aid. It does not replace referees, and its output must never be presented as though it had been through peer review.

• It reads what you give it. It cannot check a claim against an experiment it cannot run, or verify data it cannot see.
• Novelty assessment is bounded by the model's knowledge. It will miss recent or obscure prior art. Treat a novelty claim as a prompt to go and search.
• Scores are ordinal at best. Use the concerns and the revision list; treat the number as a sort key.
• Runs cost money and are non-deterministic. Two runs on the same paper will differ in wording and may differ in score.

Use it to find what you would have missed on your own reread, before a human referee finds it.

Provenance

Developed while reviewing a corpus of roughly 105 papers, and paired with citeverify for the citations reviewer. Extracted from tooling that had to work at that scale.

Development

pip install -e ".[dev]"
pytest        # 27 tests, all offline. No API key required.

Licence

MIT.