anchor-eval 0.1.0

Retrieval evaluation harness for technical corpora — span-anchored ground truth, archetype-conditioned diagnosis.

anchor-eval

Span-anchored retrieval evaluation for technical corpora.

anchor-eval measures how well your retrieval pipeline (chunking, embedding, reranking) finds the exact source spans that answer questions about your codebase or documentation. Unlike chunk-overlap metrics, span-anchored evaluation catches regressions that look fine in aggregate but silently break specific retrieval patterns.

Quickstart

pip install anchor-eval
# or: uv add anchor-eval

# 1. Generate a question set from your corpus
anchor generate --corpus ./my-repo --pack packs/code-oss-v0 --output question_set.json

# 2. Score the current pipeline
anchor score question_set.json --corpus ./my-repo

# 3. Run in CI (exit 0 = no regression, 1 = regression, 2 = corpus drift)
anchor ci --question-set question_set.json --corpus ./my-repo --baseline anchor-baseline.json

See docs/guides/quickstart.md for the full walkthrough.

The problem, explained from scratch

RAG in one sentence: Split documents into chunks → embed them → when a user asks a question, find the most similar chunks → give those chunks to an LLM.

The "documents" can be anything — PDFs, web pages, Slack messages, or code files. A .py file is a document. A .go file is a document. A code repository is just a folder of text files. The pipeline is identical.

Who actually builds RAG over code?

• Internal tools: "Ask our codebase anything" — "Where does authentication happen?", "Which function validates JWT tokens?", "What config controls the timeout?"
• Developer assistants (Cursor, Copilot workspace) — they need to retrieve the right file/function to give the model context
• Onboarding bots — new engineers asking "how does our payment flow work?"

The retriever's job in all of these: given a natural-language question, find the correct chunk of source code that answers it.

The evaluation problem

Say you want to test if your retriever is working. The obvious approach:

1. Take a chunk — say, lines 45–90 of auth.py
2. Generate a question from it: "How does this codebase validate API keys?"
3. Run retrieval and check: did you get that chunk back?

This works. Until you change your chunking strategy.

If you go from 512-token chunks to 1024-token chunks, lines 45–90 no longer exists as a chunk — it got merged into lines 1–120. Your benchmark just broke. Every question is now tied to a chunk that doesn't exist. You either throw away the benchmark or you never experiment with chunking at all.

Most teams never change chunking because they can't measure the impact. They're flying blind.

What anchor-eval does differently

Instead of anchoring a question to a chunk ID, it anchors it to a character span in the raw source file:

{
  "question": "How does this codebase validate API keys?",
  "anchor": { "file": "auth.py", "char_start": 1820, "char_end": 2140 }
}

auth.py characters 1820–2140 exist forever, regardless of how you chunk. When you run retrieval, a chunk is a "hit" if it covers that span. Change your chunking from 512 to 1024 tokens: the span is still there, the check still works, the benchmark still runs.

You can now run two configs side by side and get a real score for each. You know which one is better and why.

The diagnosis

Questions are tagged by failure type — cross_file_causality, lexical_collision, negative_space, etc. When one archetype scores 33% and another scores 90%, that pattern names the broken component. anchor-eval says "your chunker is too small for cross-file questions" rather than "retrieval is 61% overall." It then proposes the three targeted A/Bs that would confirm the cause.

What are spans?

A span is a (doc_id, char_start, char_end) triple that points to an exact range of text in a source document. Question answers in anchor-eval are defined as one or more anchor spans — the minimal text ranges that contain the answer. Hit-rate@k is computed by checking whether the retriever's top-k chunks cover those spans at a configurable IOU threshold.

Key concepts

• QuestionSet: A committed JSON file of verified questions with anchor spans and difficulty scores.
• Archetype: A failure pattern category (e.g. cross_file_causality, identifier_free_intent). Questions are tagged with archetypes so regressions appear per-category, not just in aggregate.
• Pack: A JSON bundle of archetype definitions and generator prompts. Built-in packs: code-oss-v0 (Python/TS/Go/Rust), jvm-oss-v0 (Java/Kotlin), systems-oss-v0 (Go/Rust).
• anchor ci: The CI gate. Compares current scores against a committed baseline JSON, exits 1 on per-archetype regression, exits 2 on corpus drift.

Retrieval failure archetypes

Archetype	Description
identifier_free_intent	Question uses no identifiers from the source
cross_file_causality	Answer requires reasoning across two or more files
negative_space	Correct answer is the absence of something
shadow_identifier	Multiple near-identical names compete for retrieval
lexical_collision	Non-answer spans score higher lexically than the answer
parameter_sensitivity	Answer changes depending on a numeric or enum parameter

Roadmap

v0.3 — Shard-aware generation (blocked)

v0.3 introduces corpus partitioning so large repos (50k+ files) can generate question sets in parallel shards and merge them with a CrossShardUniquenessFilter.

Why it's blocked: the filter makes a precision tradeoff — it will discard some genuinely unique questions because BM25 similarity across shards is noisier than within-shard checks. The acceptable false-positive rate is unknown until validated on a real large-scale corpus.

Gate: a public 50k-file corpus benchmark must confirm CrossShardUniquenessFilter precision before this ships. Do not implement v0.3 until that benchmark has been run and the discard rate is below an agreed threshold.

License

anchor-eval is licensed under the Business Source License 1.1. Scoring, CI, and question sets always work without a license. A license is required only for generating new question sets.

See docs/reference/licensing.md for details.