hev-rerank

Use Jev, TypeSafe's System One decision model, as a reranker. One call carries the query and up to 30 candidate documents; the model answers one question per document, in parallel: is this document relevant to the query? The answer is a calibrated probability, so the score is both a sort key (rerank) and an absolute threshold (prune).

This repo is the whole recipe: the prompt, the request state shape, a 90-line Python wrapper, and the results of running it against hosted and open-weight rerankers and LLM judges on BEIR shortlists.

Results

nDCG@10, every system permuting the same BM25 top-30 shortlist per query. Jev uses the domain-neutral phrasing in prompt.yaml, untuned per corpus.

Paired-bootstrap per query, Jev's one-state shape against each reranker: at or above Cohere on all three corpora (two wins, one tie); above Mixedbread (two wins, one interval touching zero); trading with Voyage, the strongest purpose-built reranker here (edge on SciFact, tie on NFCorpus, loss on FiQA by ~0.02); above every LLM used as a reranker on cost and latency, and on quality except Opus 5, which it ties at ~1/100 the cost.

The honest one-liner: a general decision model with no reranker training lands in the same quality and price bracket as the best purpose-built rerankers, a third of Cohere's price, and returns a calibrated probability per document that none of them do. Voyage rerank-3 is as good and slightly cheaper; what Jev adds over it is the probability and the fact that the same call shape covers routing, classification and extraction gating.

Costs are the runs' actual token usage at list price (Voyage measured from its billed usage.total_tokens). Full tables, confidence intervals, determinism, position-bias and calibration checks: RESULTS.md.

Install

pip install hev-rerank        # or: uv add hev-rerank
export TYPESAFE_API_KEY=...   # https://console.typesafe.ai/settings/keys

Use

from hev_rerank import rerank

hits = rerank(
    "does vitamin D supplementation improve bone density in older adults",
    [{"title": t, "text": x} for t, x in first_stage_results],   # strings work too
    top_n=10,          # optional
    threshold=0.1,     # optional: drop candidates the model is confident are irrelevant
)
for h in hits:
    print(h.score, h.index, h.document["title"])

rerank returns RerankResult(index, score, document) best-first. Lists longer than max_docs_per_call (30) are split into concurrent calls under Jev's ~32k-token request budget. Scores across calls are comparable because each is an absolute probability.

The shape, and why

• State: {"query": ..., "documents": {"D00": {...}, "D01": {...}, ...}} (schema).
• Questions: one Noul per document id, from prompt.yaml. All questions in a request are evaluated in parallel, so 30 documents cost about the same wall time as 5.
• Score: the Noul probability. Calibration on the SciFact run: documents scored ≥ 0.9 were judged relevant 76% of the time; documents scored < 0.1, 0.5% of the time.

Why not one Choice over the documents? It is slightly better when exactly one document is relevant (SciFact: +0.01 nDCG@10, +3 pts P@1) and much worse when many are (NFCorpus, median 16 relevant per query: 0.316 vs 0.358). A reranker that does not know the corpus should ask per-document Nouls.

Limits

• ~32k tokens per request (state + questions): ~30–50 typical passages per call.
• Rate limits are undocumented; sustained 429s appeared at ~24 requests in flight.
• Tail latency on 30-document states (p95 0.8–1.8 s) is higher than specialized hosted rerankers'. Fanning out per-pair calls trades 30× the calls for a p95 near 0.25 s per call.
• Hosted only; the data leaves your environment.

License

Apache-2.0.