refereed 0.2.0

Multi-perspective review of research papers. Open panel runs on your own model key; a calibrated panel is available as a hosted service.

refereed

Multi-perspective review of research papers. Several reviewers read the paper independently, then a synthesiser combines their reports into one recommendation.

There are two panels. Both run through the same engine, so code written against one works against the other.

	Open panel	Hosted panel
Reviewers	3 (claims, method, clarity)	7 narrow reviewers
Weights	Equal	Per-reviewer, calibrated
Calibration	None	Tuned against a corpus of ~105 papers
Runs on	Your machine	Quilio infrastructure
Model key	Yours	Included
Account	None	Quilio API key
Licence	MIT	Commercial

Install

pip install "refereed[anthropic]"

Python 3.9 or later. The engine and the open panel have no third-party dependencies; the anthropic extra is only needed to run the open panel against Claude.

Open panel

Runs locally against your own model key. No account, no rate limit.

export ANTHROPIC_API_KEY=...
refereed review paper.md
refereed reviewers            # list the panel, no API call

import asyncio
from refereed import anthropic_complete, review_paper

review = asyncio.run(review_paper(
    open("paper.md").read(),
    anthropic_complete(effort="high"),
    title="Paper title",
))

review.weighted_score
[r for r in review.reports if r.ok]
review.meta["revision_list"]

Hosted panel

Seven narrow reviewers with calibrated weights and a separate synthesiser. Requires a Quilio API key: https://quilio.dev/pricing

export QUILIO_API_KEY=qk_...
refereed review paper.md --hosted

from refereed import review

r = review(open("paper.md").read(), api_key="qk_...")
r.weighted_score
r.recommendation
r.major_concerns          # each tagged with the reviewer that raised it

QuotaExceeded is raised when the period's quota is spent. ReviewError covers transport and API failures.

Scoring

weighted_score is the weight-weighted mean of reviewer scores over reviewers that returned parseable output:

sum(weight_i * score_i) / sum(weight_i)

It is computed from the returned JSON and is reproducible from it. In the open panel the weights are equal, so this reduces to a plain mean.

Failure semantics

• A reviewer that raises, or returns output with no JSON object, is recorded with ok=False and an error string. The other reviewers still run.
• weighted_score is computed over successful reviewers only.
• Unparseable output is kept in ReviewerReport.raw, truncated to 2000 characters.
• The synthesiser runs only if at least one reviewer succeeded. A failure there sets meta_error and leaves the reports intact.
• extract_json accepts a bare object, a fenced block, or an object embedded in prose. It rejects a top-level array.

Bring your own backend

Model access goes through one callable, async (system, user) -> str. Nothing else in the open panel performs I/O.

async def complete(system: str, user: str) -> str:
    return await gateway.complete(system=system, user=user)

review = await review_paper(paper, complete)

The hosted client takes an injectable transport for the same reason.

Limits

• Assessment is bounded by the model's knowledge. It does not perform a literature search, so treat any novelty claim as a prompt to go and check.
• It reads the text you supply. It cannot run experiments or inspect data it has not been given.
• Output is non-deterministic. Two runs on the same paper differ in wording and may differ in score.
• Scores are ordinal. The concerns and the revision list carry the detail.
• Output is model-generated. It is a reading aid before submission and carries no editorial standing.

Related

• citeverify: checks whether the references in a bibliography exist, against OpenAlex, Crossref and arXiv.
• corpus-lens: keywords, rhetorical fingerprints, similarity and embeddings across a whole corpus.

citeverify output can be passed to either panel with --context.

Development

pip install -e ".[dev]"
pytest

40 tests, all offline. No model key or Quilio key required; the model backend and the HTTP transport are both injected.

Licence

MIT for this package. The hosted panel is a commercial service.