anchor-eval

Span-anchored retrieval evaluation for any text corpus.

anchor-eval measures how well your retrieval pipeline (chunking, embedding, reranking) finds the exact source spans that answer questions about your corpus — source code, documentation, legal contracts, support tickets, or any collection of text files. Unlike chunk-overlap metrics, span-anchored evaluation catches regressions that look fine in aggregate but silently break specific retrieval patterns.

Quickstart

pip install anchor-eval
# or: uv add anchor-eval
# For PDF and DOCX support:       pip install 'anchor-eval[docs]'
# For AST drift detection:        pip install 'anchor-eval[ast]'

# Scaffold a config, then generate and run CI
anchor init --domain code
anchor generate --corpus ./my-repo --domain code --output question_set.json
anchor ci --question-set question_set.json --corpus ./my-repo --baseline anchor-baseline.json

# Document corpus (Markdown, HTML, plaintext, RST)
anchor generate --corpus ./docs --domain docs --output doc_qs.json
anchor score doc_qs.json --corpus ./docs --chunking-strategy document_structure

# Try locally with Ollama (no API key required)
anchor generate --corpus ./docs --domain docs --llm-provider ollama --model llama3.2:3b

# Try with no setup at all
anchor demo
anchor demo --domain docs

--domain selects the correct pack automatically. --pack overrides it when you want a custom pack.

See docs/guides/quickstart.md for the full walkthrough.

LLM providers

anchor generate supports three providers via --llm-provider:

Provider	Flag	Key required	Notes
fake	--llm-provider fake	No	Deterministic, instant. Good for CI smoke tests.
openai	--llm-provider openai	OPENAI_API_KEY	Default model: gpt-4o-mini. Override with --model.
ollama	--llm-provider ollama	No	Runs against a local Ollama server. Default model: qwen2.5-coder:7b. Override with --model and --llm-base-url.

# Ollama on a remote host
anchor generate --corpus ./docs --domain docs \
  --llm-provider ollama --model llama3.2:3b \
  --llm-base-url http://gpu-host:11434/v1

The problem, explained from scratch

RAG in one sentence: Split documents into chunks → embed them → when a user asks a question, find the most similar chunks → give those chunks to an LLM.

The "documents" can be anything — PDFs, web pages, Slack messages, or code files. A .py file is a document. A .go file is a document. A code repository is just a folder of text files. The pipeline is identical.

Who builds RAG over text?

Almost everyone with a knowledge base. A few common shapes:

• Code corpora — "Ask our codebase anything": "Where does authentication happen?", "Which function validates JWT tokens?", "What config controls the timeout?" Developer assistants (Cursor, Copilot workspace) and onboarding bots fall here too.
• Documentation & wikis — internal runbooks, product docs, API references. Users ask in natural language; retrieval has to find the right section.
• Legal & compliance — contracts, policies, regulations. Questions have exact answers buried in specific clauses; a wrong retrieval is a liability.
• Support & ticketing — historical tickets, knowledge-base articles. Retrieval drives deflection; a missed span means a human handles it instead.

The retriever's job in all of these: given a natural-language question, find the correct chunk of text that answers it.

The evaluation problem

Say you want to test if your retriever is working. The obvious approach:

1. Take a chunk — say, lines 45–90 of auth.py
2. Generate a question from it: "How does this codebase validate API keys?"
3. Run retrieval and check: did you get that chunk back?

This works. Until you change your chunking strategy.

If you go from 512-token chunks to 1024-token chunks, lines 45–90 no longer exists as a chunk — it got merged into lines 1–120. Your benchmark just broke. Every question is now tied to a chunk that doesn't exist. You either throw away the benchmark or you never experiment with chunking at all.

Most teams never change chunking because they can't measure the impact. They're flying blind.

What anchor-eval does differently

Instead of anchoring a question to a chunk ID, it anchors it to a character span in the raw source file:

{
  "question": "How does this codebase validate API keys?",
  "anchor": { "file": "auth.py", "char_start": 1820, "char_end": 2140 }
}

auth.py characters 1820–2140 exist forever, regardless of how you chunk. When you run retrieval, a chunk is a "hit" if it covers that span. Change your chunking from 512 to 1024 tokens: the span is still there, the check still works, the benchmark still runs.

You can now run two configs side by side and get a real score for each. You know which one is better and why.

The diagnosis

Questions are tagged by failure type — cross_file_causality, lexical_collision, clause_cross_reference, etc. When one archetype scores 33% and another scores 90%, that pattern names the broken component. anchor-eval says "your chunker is splitting conditional clauses from their consequences" rather than "retrieval is 61% overall." It then proposes the targeted A/Bs that would confirm the cause.

What are spans?

A span is a (doc_id, char_start, char_end) triple that points to an exact range of text in a source document. Question answers in anchor-eval are defined as one or more anchor spans — the minimal text ranges that contain the answer. Hit-rate@k is computed by checking whether the retriever's top-k chunks cover those spans at a configurable IOU threshold.

Document spans also carry a section_id (e.g. "doc#section/page:3/p:2") for human-readable location display in reports.

Commands

Command	Description
anchor init	Scaffold an anchor.json config with domain-appropriate defaults.
anchor generate	Generate a graded QuestionSet from a corpus. Requires a license.
anchor score	Score a QuestionSet against a corpus; print hit-rate@k by archetype.
anchor run	Run a grid of retrieval configs, checkpointed to SQLite. Safe to interrupt.
anchor resume	Resume an interrupted anchor run from its checkpoint store.
anchor ci	CI gate: compare current scores against a committed baseline. Exits 0/1/2.
anchor demo	Run a 5-question demo against a built-in micro corpus. No license required.
anchor license install	Install a license token to ~/.config/anchor/license.token.
anchor license verify	Verify the installed license token and print its status.

Grid runs (anchor run / anchor resume)

anchor run executes a matrix of retrieval configs over a QuestionSet and writes a RunReport. Each config is evaluated independently so you can compare chunking strategies, retrieval modes, and k-values side by side.

# Single default config (BM25, fixed-token chunking)
anchor run --question-set qs.json --corpus ./my-repo --output report.json

# Custom grid from a JSON config file
anchor run --question-set qs.json --corpus . --grid-config grid.json --output report.json

# Estimate cost without running
anchor run --question-set qs.json --corpus . --estimate-only

# Resume after Ctrl-C
anchor resume --question-set qs.json --corpus . --store run.db --output report.json

The checkpoint store (run.db) is a WAL-mode SQLite file. Each RunUnit (question × config) is claimed and completed atomically, so interrupted runs restart from exactly where they left off.

Large-document handling

anchor-eval handles large documents — books, long legal contracts, multi-page PDFs, enormous Markdown files — without dropping coverage or blowing up LLM context budgets.

The problem: a 100-page PDF may have one giant "section" spanning the entire document. Previous versions either kept it as a single span (too large for any LLM prompt) or silently skipped it (losing all questions from that content). Neither is acceptable.

What anchor-eval does: when a naturally-selected region exceeds MAX_SPAN_CHARS, the extractor splits it into overlapping sliding-window sub-spans instead of emitting a single oversized span or dropping it. Each sub-span:

• Is at most MAX_SPAN_CHARS characters (default 2 000)
• Overlaps the previous by SPAN_WINDOW_OVERLAP_CHARS characters (default 200, ~10 %) so content at boundaries is never missed
• Snaps its end to the nearest word boundary so spans never cut mid-token
• Must be at least MIN_SPAN_CHARS characters to be emitted — the minimum-size contract is enforced per sub-window, not just for the region as a whole

This applies to all prose and document extractors: Markdown sections, HTML blocks, PDF paragraphs, plaintext/RST/DOCX paragraphs.

To prevent cost explosion on very large documents (after sliding-window expansion, a 200k-char document could produce hundreds of candidate spans), SpanSelector caps output at MAX_SPANS_PER_DOC = 20 spans per archetype per document. The cap uses a deterministic, per-(doc, archetype) random sample so each archetype gets an independent selection and coverage is spread across the document rather than biased to its head.

Tunable defaults (override in Python or via future CLI flags):

Constant	Default	What it controls
MAX_SPAN_CHARS	2000	Maximum chars per anchor span (window size for large-doc splits)
SPAN_WINDOW_OVERLAP_CHARS	200	Overlap between consecutive sliding-window sub-spans
MAX_SPANS_PER_DOC	20	Max candidate spans per document per archetype after expansion
MIN_SPAN_CHARS	30	Minimum chars for any span or sub-window to be emitted

import anchor.defaults as d
d.MAX_SPAN_CHARS = 3000           # longer windows for dense legal prose
d.SPAN_WINDOW_OVERLAP_CHARS = 300 # more overlap for tightly-coupled clauses
d.MAX_SPANS_PER_DOC = 30          # more coverage on large docs, higher cost

AST spans and drift detection (--ast)

Pass --ast to anchor generate to enrich spans with tree-sitter AST node IDs. This enables structural drift detection: if a function is renamed or moved, anchor drift detects it and flags the affected questions before a CI run wastes time on stale anchors.

# Requires: pip install 'anchor-eval[ast]'  and a 'generate:ast' license entitlement
anchor generate --corpus ./my-repo --domain code --ast --output qs.json

AST enrichment is off by default because it adds tree-sitter parsing overhead and requires the ast optional dependency.

Key concepts

• QuestionSet: A committed JSON file of verified questions with anchor spans and difficulty scores.
• Archetype: A failure pattern category tagged on each question. Regressions appear per-archetype, not just in aggregate.
• Pack: A JSON bundle of archetype definitions and generator prompts for a specific domain.
• anchor ci: The CI gate. Compares current scores against a committed baseline JSON, exits 1 on per-archetype regression, exits 2 on corpus drift.
• anchor run: A checkpoint-based grid runner. Evaluates multiple retrieval configs in one pass and writes a structured RunReport.
• RunStore: SQLite-backed checkpoint store. Each unit (question × config) is claimed atomically so concurrent or resumed runs are safe.

Built-in packs

Pack	Domain	Archetypes
code-oss-v0	Python, TS, Go, Rust	identifier_free_intent, cross_file_causality, negative_existence, shadow_identifier, lexical_collision, parameter_level
jvm-oss-v0	Java, Kotlin	identifier_free_intent, cross_file_causality, parameter_level, overload_disambiguation, annotation_sensitivity, generic_type_boundary
systems-oss-v0	Go, Rust	identifier_free_intent, cross_file_causality, ownership_borrow_context, unsafe_block_scope, trait_impl_dispatch, cgo_ffi_boundary
docs-general-v0	Markdown, HTML, RST, plaintext	section_cross_reference, implicit_negative, term_definition_lookup, conditional_answer, table_cell_lookup, procedural_step
docs-legal-v0	Legal contracts, regulations	clause_cross_reference, effective_date_shadowed, defined_term_collision, implicit_obligation, negative_obligation, jurisdiction_qualifier
docs-support-v0	Support tickets, runbooks	symptom_cause_link, workaround_vs_fix, version_specific_answer, escalation_path, product_name_alias

Custom packs can be written for any domain — define archetypes, write prompts with {text_excerpt} / {archetype_description} / {output_format}, and point --pack at the directory.

anchor-corpus.json manifest

Drop an anchor-corpus.json at the root of any corpus directory to configure domain defaults:

{
  "domain": "legal",
  "version": "2024-q4",
  "description": "ACME Corp master service agreement corpus, 2024 edition"
}

The loader merges domain, version, and description into every SourceDocument.metadata and activates domain-specific chunking (e.g. numbered-section splitting for "domain": "legal").

Note (v0.2.2): only the three keys above are forwarded into document metadata. Custom fields are ignored to prevent unintended data leakage.

Supported file types

Extension	Content type	Notes
.py	code_python
.ts, .tsx	code_typescript
.js, .jsx, .mjs	code_javascript
.java	code_java
.kt, .kts	code_kotlin
.cs	code_csharp
.go	code_golang
.rs	code_rust
.md	docs_markdown
.txt, .log	docs_plaintext
.html, .htm	docs_html	Tags stripped; plain text stored
.rst	docs_rst
.pdf	docs_pdf	Requires anchor-eval[docs]
.docx	docs_docx	Requires anchor-eval[docs]
.yaml, .yml, .json	docs_openapi

Changelog

v0.3.1

Large-document handling and generation correctness:

• Sliding-window span generation — prose and document extractors (Markdown, HTML, PDF, plaintext/RST/DOCX) now split sections that exceed MAX_SPAN_CHARS into overlapping sub-spans instead of emitting a single oversized span or dropping the content. No coverage is lost on large documents regardless of how they are structured.
• SPAN_WINDOW_OVERLAP_CHARS (default 200) — configurable overlap between consecutive sub-spans so content at window boundaries is never missed.
• MAX_SPANS_PER_DOC (default 20) — per-(document, archetype) cap on candidate spans after sliding-window expansion. Uses a deterministic, archetype-seeded random sample so each archetype gets an independent selection. Without this, a single 100k-char document could produce hundreds of candidates and multiply LLM call cost by 10×.
• Anti-leakage gate fix — the Jaccard leakage check now compares against the same span text the LLM was shown (capped at MAX_PROMPT_SPAN_CHARS). Previously, for spans over 1 500 chars, tail tokens the LLM never saw inflated the Jaccard denominator and allowed genuinely leaking questions to pass the gate.
• Anchor integrity fix — MAX_PROMPT_SPAN_CHARS is now set equal to MAX_SPAN_CHARS so the anchor recorded in the question set (char_start/char_end) always covers exactly what the LLM saw when generating the question. A gap between these two values would produce questions that are unanswerable from their own anchors.
• Sub-window minimum-size fix — _sliding_window_spans now enforces the caller's min_chars per sub-window. Previously, the last sub-window could be shorter than _MIN_HTML_BLOCK_CHARS (300) because the guard only applied to the overall region, not to individual windows produced by the splitter.

v0.3.0

New features:

• anchor corpus readiness — corpus health scanner (parse failures, near-duplicates, coverage gaps, table anomalies)
• anchor run --trajectories — score agentic tool-call sequences alongside span recall
• anchor merge-sets — merge question sets from multiple packs or corpus subdirectories
• anchor drift query — vocabulary drift detection (BM25 zero-result check)
• anchor drift doc — document section drift detection (section_id re-anchoring check)
• docs-agentic-v0 pack with 5 trajectory archetypes (tool_call_abstention, tool_call_hallucination, tool_selection_error, parameter_grounding, loop_nontermination)
• Dual-ground-truth question schema: questions can carry both anchors (span) and expected_trajectory (agentic) — at least one required
• schema_version "1.1" auto-set by anchor generate when any question has expected_trajectory
• RunReport.latency_metrics: P50/P95/P99 latency, quality-per-second, quality-per-GPU-hour
• Archetype.trajectory_archetype flag; GenerationPipeline now synthesizes hybrid questions for trajectory archetypes

v0.2.2

Security:

• anchor generate --ast now uses full ed25519 verification for the generate:ast entitlement — previously the second gate fell into placeholder-key mode and accepted any non-empty token
• License server: email plain-text body strips newlines from the customer org name, preventing RFC 2822 header injection
• License server: ANCHOR_SIGNING_KEY_HEX is validated to be exactly 64 hex characters before being passed to nacl

Performance:

• Dense retrieval (_cosine_top_k) uses numpy.dot when available — ~300–1000× faster than the previous Python loop for real embedding dimensions (768–3072)
• UniquenessVerifier default top_k raised from 5 → 20, catching ambiguous questions in larger corpora that were previously missed

Correctness:

• EmbeddingCache.get_or_compute is now thread-safe (double-checked lock pattern)
• engine_version in generated QuestionSet files now reflects the actual installed package version, fixing false engine-mismatch exits from anchor ci
• OpenSearch _execute_search skips malformed hits (missing _source fields) with a warning instead of crashing the scoring run
• PDF and DOCX extraction failures (corrupt files, bad ZIP, etc.) are now logged and skipped rather than aborting the entire corpus load
• LocalTelemetryEmitter handles read-only filesystems gracefully; telemetry file rotates at 10 MB

Configuration:

• OpenSearchIndexBackend accepts http_auth, use_ssl, and verify_certs constructor parameters, with automatic fallback to OPENSEARCH_USERNAME/OPENSEARCH_PASSWORD environment variables

Breaking change — anchor-corpus.json metadata forwarding: Custom string fields in anchor-corpus.json beyond domain, version, and description are no longer forwarded into SourceDocument.metadata. If you relied on a custom field (e.g. "project") appearing in document metadata, add it under one of the three allowlisted keys or access it from the manifest directly.

v0.2.1

• Ollama LLM provider (--llm-provider ollama)
• Tree-sitter AST adapters for Python, TypeScript, Go, Rust (--ast)
• PDF and DOCX corpus support (anchor-eval[docs])
• Production ed25519 license key deployed; Svix replay protection via SQLite
• HTML report XSS protection

v0.2.0

• anchor run / anchor resume grid runner with SQLite checkpoint store
• anchor drift semantic drift detection
• anchor-corpus.json manifest
• JVM pack (jvm-oss-v0) and Systems pack (systems-oss-v0)
• Docs packs: docs-general-v0, docs-legal-v0, docs-support-v0

Roadmap

v0.3.1 (shipped)

• Sliding-window span generation — large documents no longer lose coverage; oversized sections are split into overlapping sub-spans with configurable window size and overlap
• Anti-leakage and anchor integrity fixes — leakage check now uses the same token set the LLM was shown; anchor spans always match the LLM's view of the text

v0.3.0 (shipped)

• Corpus readiness diagnostics — anchor corpus readiness scans for parse failures, near-duplicate documents, coverage gaps, and DOCX/PDF table anomalies before generation
• Efficiency metrics — RunReport now includes latency_metrics (P50/P95/P99 latency, quality-per-second, quality-per-GPU-hour)
• Agentic / trajectory evaluation — docs-agentic-v0 pack with 5 trajectory archetypes; anchor run --trajectories scores tool-call sequences alongside span recall; dual-ground-truth question schema (span anchor + ExpectedTrajectory)
• anchor merge-sets — merge question sets from different corpus subdirectories or packs
• anchor drift query and anchor drift doc — vocabulary drift and document section drift detection
• schema_version 1.1 — automatically set by anchor generate when trajectory archetypes produce ExpectedTrajectory fields

v0.4 — Shard-aware generation (planned)

v0.4 introduces corpus partitioning so large repos (50k+ files) can generate question sets in parallel shards and merge them with a CrossShardUniquenessFilter.

Why it's blocked: the filter makes a precision tradeoff — it will discard some genuinely unique questions because BM25 similarity across shards is noisier than within-shard checks. The acceptable false-positive rate is unknown until validated on a real large-scale corpus.

Gate: a public 50k-file corpus benchmark must confirm CrossShardUniquenessFilter precision before this ships.

License

anchor-eval is licensed under the Business Source License 1.1. Scoring, CI, and question sets always work without a license. A license is required only for generating new question sets.

Pricing: $49 / user / month · $149 / user / year — see licensing details.