notorious 0.0.1

Build a retrieval eval for your own corpus: pool candidates, LLM-judge them under a rubric you own, and measure judge-vs-human alignment before trusting any metric.

Notorious NDCG

Did that change to your search actually make it better? Answer it with an LLM-graded answer key you've checked you can trust — then compare your retrieval configurations using standard search metrics.

Why you'd use this

You changed something in your search or RAG stack — swapped the embedding model, resized your chunks, added a reranker, retuned your hybrid-search weights. The question is always the same:

Did retrieval actually get better, or did the numbers just move around?

Most teams answer by eyeballing a handful of queries. That doesn't scale, and it quietly misses regressions: a change that helps ten queries and hurts twenty can still "look fine."

Answering it properly needs an answer key — for a set of test queries, which documents in your corpus are actually relevant. With that key you can compute real search-quality scores and compare configurations head to head. The catch: a good answer key is expensive to build by hand, and the obvious shortcut — "just ask an LLM which documents are relevant" — trades one problem for another. Does the LLM's notion of relevant match yours? If you can't answer that, every metric built on top of it is standing on sand.

notorious is built around closing exactly that gap:

1. It uses LLMs to grade relevance at scale, so you don't hand-label thousands of query–document pairs.
2. Before trusting those grades, it makes the agreement measurable. You review a small sample, correct the grades you disagree with, and the tool reports how often the judge matched you — with a confidence interval, so you know when you've checked enough.
3. You tune a rubric (your written definition of "relevant") until the judge reproduces your corrections on its own. Then you freeze it and score the full test set.

What you get out: a portable TREC-style qrels file (the answer key) and, per configuration, NDCG@k, Recall@k, and MRR with error bars across multiple judge models. Everything is plain files on disk you can open, diff, and re-run.

Those metric names are opaque until someone tells you what they mean:

• NDCG@k — of the top k results, are the most-relevant ones near the top? (overall ranking quality)
• Recall@k — of all the relevant documents, how many made it into the top k?
• MRR — on average, how high up the list is the first relevant result?

Install

pip install notorious          # or: uv pip install notorious
notorious --help

Prefer an isolated tool install:

uvx notorious --help           # run without installing
pipx install notorious         # or install it as a standalone CLI

From source (for development)

git clone https://github.com/georgian-io/notorious
cd notorious
uv sync                        # create the environment
uv run notorious --help

Quickstart

The scaffold ships mock, deterministic judge models, so you can run the entire loop offline and free — before spending a cent or wiring up your own search:

notorious init my-eval        # scaffold a runnable project: toy corpus, rubric, configs, mock judges
notorious run my-eval --yes   # pool → judge → report, all at once
notorious review my-eval      # open the browser page to grade the judge

What to expect:

• The mock judges need no API key and cost nothing; every artifact lands in my-eval/out/.
• review is keyboard-first and blind — the judge's grade reveals only after you commit yours, which is what keeps the agreement number honest.
• Lost the thread? notorious status my-eval prints the current state and the exact next command to run. It never writes anything, so run it any time.

New to the idea? The walkthrough narrates this same run end to end with screenshots. Ready for your own stack? Skip to Bringing your own corpus and configs.

How it works

The four stages

There are four stages. Each one reads and writes a file on disk you can open, hand-edit, diff, or re-run on its own:

queries.jsonl ──► pool ──► judge ──► report

pool — for every test query, runs each of your configs and collects the union of their top-k results into one candidate set (out/pool.jsonl). This is pooling: rather than grade your entire corpus against every query (impossible at any real scale), you grade only the documents your configs actually surface. Deduped by doc_id, with each config's rank recorded.

judge — asks each judge model to grade every (query, document) pair in the pool on a 0–3 relevance scale, with a written rationale, following your rubric.md. Grades are cached by content, so a re-run only calls the model for genuinely new pairs and an interrupted run resumes with no lost work. Calls run concurrently (judge_concurrency, with an optional global judge_rate_limit_per_min).

The prompt treats the query and document as untrusted data — the model is told to ignore any instructions embedded in them, and delimiter tags inside the text are defanged — so a document can't talk the judge into a grade.

report — turns the grades into the answer key and the metrics. It writes:

• out/qrels.txt — the TREC-style answer key;
• per-config NDCG@k / Recall@k / MRR — plus P@k as a diagnostic (exact, not pool-relative, since every top-k doc is judged);
• out/report.md — a readable summary, led by a decision box (run status, judge trust, comparison verdict, the primary blocker, and the one next action);
• out/report.json — the same metrics, significance, alignment, and diagnostics as stable JSON, for CI gates and dashboards (no prose scraping);
• out/review.html — the interactive review page.

With multiple judges, every metric carries the spread across them as error bars. report also runs a paired per-query significance test between every pair of configs under test — the mean NDCG@k difference with a bootstrap confidence interval — so you can tell a real win from query-sampling noise, which is the whole "did it actually get better?" question. A comparison whose interval is too wide to decide is flagged inconclusive (rather than a tie), which is the signal to judge more queries — a small inner-loop sample earns judge trust, but the decision usually wants the larger frozen run.

review — serves the review page from a local, loopback-only server (http://127.0.0.1:8765), rendered live from the current files. This is where you grade the judge card by card (see Aligning the judge). Grading is keyboard-first — 0–3 to grade, ←/→ to move, u to jump to the next ungraded card — and a session panel tracks your graded count, grade distribution, and relevant-vs-not balance as you go.

Every run records its own provenance

Each of pool, judge, and report also writes out/run-manifest.json — a provenance sidecar recording the tool version, hashes of every input (run.yaml, rubric, queries, configs, pool), the judge set, the judgment split, and when each stage last ran. It complements the human-readable report: it's the one file that answers "what exactly produced these numbers?" without reconstructing it from the others.

Convenience and inspection commands

• notorious run chains pool → judge → report in one go, with the same cost confirmation as judge (--yes skips it). It stops on the first stage that errors and prints the next human action. It's an on-ramp, not a replacement — every stage stays independently runnable, which is what the inner/outer loop relies on.
• notorious status is a read-only inspector for when you've lost track of where you are. It prints which artifacts exist, the current rubric hash and query-provenance mix, how the pool lines up with your queries, how many judge calls a judge run would make now (and how many are cached), the standing corrections and their alignment CI — and, from all of that, the single command to run next. It reuses the pipeline's own bookkeeping, so its counts are exactly what the next stage will act on. Run it any time; it never writes anything.
• notorious doctor preflights the configured judge models and query generator offline before a real (paid) run — no calls made. For each model it reports whether its API key is visible to LiteLLM and whether LiteLLM knows its price, and exits non-zero if any model is missing a key. Mock models are always ready.
• notorious validate is the retrieval-side counterpart to doctor: it imports your configs.py and calls each config's search on a few queries (no LLM calls), checking every result is a well-formed (doc_id, text) list with unique, non-empty ids — and reports each config's latency and a few sample ids. It exits non-zero on a malformed adapter, so a wiring bug surfaces here instead of mid-pool.

The trust loop — earn it, then spend it

The heart of the tool is a two-loop workflow:

• Inner loop (small, cheap, interactive). On a sample of ~20–50 queries, grade cards blind against your rubric, watch how often the judge agrees — with a confidence interval — and edit rubric.md until agreement is high and the interval is tight. This earns trust in the judge.
• Outer loop (large, frozen, repeatable). Freeze the rubric, point at the full query set, and score every config with the trusted ensemble. This spends that trust on a decision — and you re-run it on any change (new embedding model, reranker, chunking tweak) to watch the metrics move.

→ Aligning the judge walks through both loops in detail: grading blind, the alignment score's two levers (interval width = "reviewed enough?", level = "does the judge agree with me?"), why you need two-sided corrections, and the ensemble-transfer check.

Reading the report

The report leads with the decision: for every pair of configs under test, is one a real winner, a genuine tie, or inconclusive (underpowered — go judge more queries)? Backing that up are three numbers that all look like error bars but answer three different questions, computed with different math on purpose — treating them alike is the most common way to misread a report.

→ Reading the numbers explains all three (cross-judge spread, the Wilson alignment interval, the bootstrap significance interval) and why a tie is a decision while inconclusive is a to-do.

Bringing your own corpus and configs

The tool never ingests, embeds, chunks, or indexes anything — you keep your retrieval stack and expose each configuration you want to compare as a small adapter. A "config" is any object with a search method that takes a query and returns ranked (doc_id, text) pairs, best result first:

# my-eval/configs.py
class EmbeddingSearch:
    def __init__(self, model): ...
    def search(self, query: str) -> list[tuple[str, str]]:
        # call your vector DB / search service; return results best-first
        return [("doc-42", "text of doc 42"), ("doc-7", "text of doc 7")]

# Each entry is one configuration you're comparing — a "row" in the report.
CONFIGS = {
    "embed-large": EmbeddingSearch("text-embedding-3-large"),
    "embed-small": EmbeddingSearch("text-embedding-3-small"),
}

A config can be anything behind that method — a raw embedding search, hybrid fusion, a reranker stacked on a retriever, query rewriting. List which configs to compare in run.yaml, and point judge_models at real LLMs (e.g. openai/gpt-4.1, anthropic/claude-sonnet-5, gemini/gemini-2.5-pro — three are recommended, as above).

No test queries yet, but you have a corpus? Bootstrap some.

uv run notorious generate-queries my-eval --chunks corpus.jsonl --per-chunk 2

This drafts questions each chunk could answer (via query_gen_model), tagged generated. By default they land in a separate queries.generated.jsonl — a draft to review, not answer-key evidence — leaving your eval set untouched; pass --append to fold them straight into queries.jsonl, or --out FILE to name a target. It shows a cost estimate (with the same confirmation as judge) before calling, and drops near-duplicate drafts.

Once promoted, generated queries flow through the same pool → judge path as real ones — the source chunk is never treated as the answer key — and every report prints the provenance mix, so you can see how much of your eval rests on generated vs. real queries.

Reviewing without the local server

Corrections can also be saved without notorious review: open the static out/review.html as a plain file, and Save uses the File System Access API where available, or downloads a corrections file that notorious merge-corrections my-eval corrections.jsonl folds back into the standing set.

Which files carry state

Both loops are the same pipeline over different query sets. In short: queries.jsonl is the working set (whatever it holds is what gets pooled, judged, and reported); out/pool.jsonl is derived and disposable; and the durable, content-addressed stores are out/corrections.jsonl (your ground truth, which survives rubric edits) and out/judgments.jsonl (the judge cache). Corrections are single-reviewer, last-write-wins.

→ Queries and artifacts has the full rules: sampling an inner-loop subset, cache maintenance (cache stats|compact|prune), the single-reviewer model, and why a rubric edit re-judges the whole pool.

What's included

The full pipeline is complete and hardened for honest comparisons:

• The core loop — pool → judge → report, plus the interactive review/alignment page and multi-judge ensembles that report every metric as mean ± spread across judges.
• Comparison-validity guards — the pool stage refuses when two configs return the same doc_id with different text (they must retrieve from the same corpus snapshot), and chunking experiments can declare parent documents via structured ids (id_separator) instead of a hand-maintained parent_map.
• Diagnostics in every report — a pool-diversity measure (are your configs blind in the same way?), a grade-concentration tripwire that flags when the 0–3 scale has stopped separating your configs, and paired per-query significance (with a bootstrap CI) on the NDCG gap between every two configs under test, flagging a win, a genuine tie, or an inconclusive (underpowered) comparison that needs more queries.
• Pool wideners — pool_only configs (e.g. a lexical/BM25 retriever) broaden the candidate set so a pool of similar configs can't look falsely complete. init writes an in-memory BM25 example into your configs module; point it at your own server-side lexical search at scale. When a widener is present, the report adds an experimental pool-completeness estimate — capture-recapture (Lincoln–Petersen) over how much the dense configs and the lexical widener overlap on relevant docs, giving a rough "what fraction of the relevant population is in the pool?" with a CI. It's a heuristic with stated assumptions, offered alongside the pool-relative recall caveat, never as a replacement for it.
• Practical judging — concurrent calls with an optional global rate limit, content-addressed caching that resumes interrupted runs, and an optional query generator (generate-queries) that drafts queries from corpus chunks.

Judging is pointwise — each pair graded independently on 0–3. That is the only mode implemented; a comparison-based (pairwise) mode is a possible future direction, not a shipped feature, and would take more than a new labeler subclass (the pipeline plans, caches, and costs one pair at a time). The grade-concentration diagnostic is what would tell you, on evidence, that this corpus needs one.

Documentation

Deeper guides live in docs/:

• Aligning the judge — the inner/outer trust loop in detail.
• Reading the numbers — the three intervals and the decision they support.
• Queries and artifacts — which files carry state; cache maintenance.
• The report.json contract — machine-readable report, JSON Schema, CI-gating recipes.
• Examples — runnable projects, from an offline walkthrough to real-corpus benchmarks.
• Development — set up the environment, run the checks, build the package.
• Contributing — conventions for pull requests.