refereed

Multi-perspective review of research papers. Several reviewers read the paper independently, then a synthesiser combines their reports into one recommendation.

There are two panels. Both run through the same engine, so code written against one works against the other.

Install

pip install "refereed[anthropic]"

Python 3.9 or later. The engine and the open panel have no third-party dependencies; the anthropic extra is only needed to run the open panel against Claude.

Open panel

Runs locally against your own model key. No account, no rate limit.

export ANTHROPIC_API_KEY=...
refereed review paper.md
refereed reviewers            # list the panel, no API call

import asyncio
from refereed import anthropic_complete, review_paper

review = asyncio.run(review_paper(
    open("paper.md").read(),
    anthropic_complete(effort="high"),
    title="Paper title",
))

review.weighted_score
[r for r in review.reports if r.ok]
review.meta["revision_list"]

Hosted panel

Seven narrow reviewers with calibrated weights and a separate synthesiser. Requires a Quilio API key: https://quilio.dev/pricing

export QUILIO_API_KEY=qk_...
refereed review paper.md --hosted

from refereed import review

r = review(open("paper.md").read(), api_key="qk_...")
r.weighted_score
r.recommendation
r.major_concerns          # each tagged with the reviewer that raised it

QuotaExceeded is raised when the period's quota is spent. ReviewError covers transport and API failures.

Scoring

weighted_score is the weight-weighted mean of reviewer scores over reviewers that returned parseable output:

sum(weight_i * score_i) / sum(weight_i)

It is computed from the returned JSON and is reproducible from it. In the open panel the weights are equal, so this reduces to a plain mean.

Failure semantics

• A reviewer that raises, or returns output with no JSON object, is recorded with ok=False and an error string. The other reviewers still run.
• weighted_score is computed over successful reviewers only.
• Unparseable output is kept in ReviewerReport.raw, truncated to 2000 characters.
• The synthesiser runs only if at least one reviewer succeeded. A failure there sets meta_error and leaves the reports intact.
• extract_json accepts a bare object, a fenced block, or an object embedded in prose. It rejects a top-level array.

Bring your own backend

Model access goes through one callable, async (system, user) -> str. Nothing else in the open panel performs I/O.

async def complete(system: str, user: str) -> str:
    return await gateway.complete(system=system, user=user)

review = await review_paper(paper, complete)

The hosted client takes an injectable transport for the same reason.

Limits

• Assessment is bounded by the model's knowledge. It does not perform a literature search, so treat any novelty claim as a prompt to go and check.
• It reads the text you supply. It cannot run experiments or inspect data it has not been given.
• Output is non-deterministic. Two runs on the same paper differ in wording and may differ in score.
• Scores are ordinal. The concerns and the revision list carry the detail.
• Output is model-generated. It is a reading aid before submission and carries no editorial standing.

Related

• citeverify: checks whether the references in a bibliography exist, against OpenAlex, Crossref and arXiv.
• corpus-lens: keywords, rhetorical fingerprints, similarity and embeddings across a whole corpus.

citeverify output can be passed to either panel with --context.

Development

pip install -e ".[dev]"
pytest

40 tests, all offline. No model key or Quilio key required; the model backend and the HTTP transport are both injected.

Licence

MIT for this package. The hosted panel is a commercial service.