anchor-eval 0.2.1

Retrieval evaluation harness for technical corpora — span-anchored ground truth, archetype-conditioned diagnosis.

anchor-eval

Span-anchored retrieval evaluation for any text corpus.

anchor-eval measures how well your retrieval pipeline (chunking, embedding, reranking) finds the exact source spans that answer questions about your corpus — source code, documentation, legal contracts, support tickets, or any collection of text files. Unlike chunk-overlap metrics, span-anchored evaluation catches regressions that look fine in aggregate but silently break specific retrieval patterns.

Quickstart

pip install anchor-eval
# or: uv add anchor-eval
# For PDF and DOCX support: pip install 'anchor-eval[docs]'

# Code corpus — generate a question set, then run CI
anchor generate --corpus ./my-repo --domain code --output question_set.json
anchor ci --question-set question_set.json --corpus ./my-repo --baseline anchor-baseline.json

# Document corpus (Markdown, HTML, plaintext, RST)
anchor generate --corpus ./docs --domain docs --output doc_qs.json
anchor score doc_qs.json --corpus ./docs --chunking-strategy document_structure

# Legal corpus
anchor generate --corpus ./contracts --domain legal --output legal_qs.json

# Try with no setup
anchor demo
anchor demo --domain docs

--domain selects the correct pack automatically. --pack overrides it when you want a custom pack.

See docs/guides/quickstart.md for the full walkthrough.

The problem, explained from scratch

RAG in one sentence: Split documents into chunks → embed them → when a user asks a question, find the most similar chunks → give those chunks to an LLM.

The "documents" can be anything — PDFs, web pages, Slack messages, or code files. A .py file is a document. A .go file is a document. A code repository is just a folder of text files. The pipeline is identical.

Who builds RAG over text?

Almost everyone with a knowledge base. A few common shapes:

• Code corpora — "Ask our codebase anything": "Where does authentication happen?", "Which function validates JWT tokens?", "What config controls the timeout?" Developer assistants (Cursor, Copilot workspace) and onboarding bots fall here too.
• Documentation & wikis — internal runbooks, product docs, API references. Users ask in natural language; retrieval has to find the right section.
• Legal & compliance — contracts, policies, regulations. Questions have exact answers buried in specific clauses; a wrong retrieval is a liability.
• Support & ticketing — historical tickets, knowledge-base articles. Retrieval drives deflection; a missed span means a human handles it instead.

The retriever's job in all of these: given a natural-language question, find the correct chunk of text that answers it.

The evaluation problem

Say you want to test if your retriever is working. The obvious approach:

1. Take a chunk — say, lines 45–90 of auth.py
2. Generate a question from it: "How does this codebase validate API keys?"
3. Run retrieval and check: did you get that chunk back?

This works. Until you change your chunking strategy.

If you go from 512-token chunks to 1024-token chunks, lines 45–90 no longer exists as a chunk — it got merged into lines 1–120. Your benchmark just broke. Every question is now tied to a chunk that doesn't exist. You either throw away the benchmark or you never experiment with chunking at all.

Most teams never change chunking because they can't measure the impact. They're flying blind.

What anchor-eval does differently

Instead of anchoring a question to a chunk ID, it anchors it to a character span in the raw source file:

{
  "question": "How does this codebase validate API keys?",
  "anchor": { "file": "auth.py", "char_start": 1820, "char_end": 2140 }
}

auth.py characters 1820–2140 exist forever, regardless of how you chunk. When you run retrieval, a chunk is a "hit" if it covers that span. Change your chunking from 512 to 1024 tokens: the span is still there, the check still works, the benchmark still runs.

You can now run two configs side by side and get a real score for each. You know which one is better and why.

The diagnosis

Questions are tagged by failure type — cross_file_causality, lexical_collision, clause_cross_reference, etc. When one archetype scores 33% and another scores 90%, that pattern names the broken component. anchor-eval says "your chunker is splitting conditional clauses from their consequences" rather than "retrieval is 61% overall." It then proposes the targeted A/Bs that would confirm the cause.

What are spans?

A span is a (doc_id, char_start, char_end) triple that points to an exact range of text in a source document. Question answers in anchor-eval are defined as one or more anchor spans — the minimal text ranges that contain the answer. Hit-rate@k is computed by checking whether the retriever's top-k chunks cover those spans at a configurable IOU threshold.

Document spans also carry a section_id (e.g. "doc#section/page:3/p:2") for human-readable location display in reports.

Key concepts

• QuestionSet: A committed JSON file of verified questions with anchor spans and difficulty scores.
• Archetype: A failure pattern category tagged on each question. Regressions appear per-archetype, not just in aggregate.
• Pack: A JSON bundle of archetype definitions and generator prompts for a specific domain.
• anchor ci: The CI gate. Compares current scores against a committed baseline JSON, exits 1 on per-archetype regression, exits 2 on corpus drift.

Built-in packs

Pack	Domain	Archetypes
code-oss-v0	Python, TS, Go, Rust	identifier_free_intent, cross_file_causality, negative_existence, shadow_identifier, lexical_collision, parameter_level
jvm-oss-v0	Java, Kotlin	identifier_free_intent, cross_file_causality, parameter_level, overload_disambiguation, annotation_sensitivity, generic_type_boundary
systems-oss-v0	Go, Rust	identifier_free_intent, cross_file_causality, ownership_borrow_context, unsafe_block_scope, trait_impl_dispatch, cgo_ffi_boundary
docs-general-v0	Markdown, HTML, RST, plaintext	section_cross_reference, implicit_negative, term_definition_lookup, conditional_answer, table_cell_lookup, procedural_step
docs-legal-v0	Legal contracts, regulations	clause_cross_reference, effective_date_shadowed, defined_term_collision, implicit_obligation, negative_obligation, jurisdiction_qualifier
docs-support-v0	Support tickets, runbooks	symptom_cause_link, workaround_vs_fix, version_specific_answer, escalation_path, product_name_alias

Custom packs can be written for any domain — define archetypes, write prompts with {text_excerpt} / {archetype_description} / {output_format}, and point --pack at the directory.

anchor-corpus.json manifest

Drop an anchor-corpus.json at the root of any corpus directory to configure domain defaults:

{
  "domain": "legal",
  "language": "en",
  "description": "ACME Corp master service agreement corpus, 2024 edition"
}

The loader merges these fields into every SourceDocument.metadata and activates domain-specific chunking (e.g. numbered-section splitting for "domain": "legal").

Supported file types (v0.2.1)

Extension	Content type	Notes
.py	code_python
.ts, .tsx	code_typescript
.js, .jsx, .mjs	code_javascript
.java	code_java
.kt, .kts	code_kotlin
.cs	code_csharp
.go	code_golang
.rs	code_rust
.md	docs_markdown
.txt, .log	docs_plaintext
.html, .htm	docs_html	Tags stripped; plain text stored
.rst	docs_rst
.pdf	docs_pdf	Requires anchor-eval[docs]
.docx	docs_docx	Requires anchor-eval[docs]
.yaml, .yml, .json	docs_openapi

Roadmap

v0.3 — Shard-aware generation (blocked)

v0.3 introduces corpus partitioning so large repos (50k+ files) can generate question sets in parallel shards and merge them with a CrossShardUniquenessFilter. Also includes anchor merge-sets to combine question sets from different corpus subdirectories.

Why it's blocked: the filter makes a precision tradeoff — it will discard some genuinely unique questions because BM25 similarity across shards is noisier than within-shard checks. The acceptable false-positive rate is unknown until validated on a real large-scale corpus.

Gate: a public 50k-file corpus benchmark must confirm CrossShardUniquenessFilter precision before this ships.

License

anchor-eval is licensed under the Business Source License 1.1. Scoring, CI, and question sets always work without a license. A license is required only for generating new question sets.

Pricing: $49 / user / month · $149 / user / year — see licensing details.