hitgate

Evaluate your RAG retrieval without labeled datasets — a regression-gated quality harness for retrieval ranking, built around a small hybrid retriever. Measure whether a change helps or hurts when you have no labels, no users to A/B against, and no budget for manual annotation.

The Problem

Building RAG systems is straightforward. Knowing whether a change made retrieval better or worse is hard — especially with a small corpus and no production telemetry. You can't A/B test with one user. You can't afford hand-labeled golden sets. LLM judges are expensive and opaque.

hitgate solves one specific problem: detect retrieval regressions label-free. Like a test suite for ranking quality.

What hitgate Does (and Doesn't)

✓ Proves whether retrieval improved or regressed between two runs (regression detection)
✓ Works with zero labeled data — no golden sets required to start
✓ Gates on measurable deltas — reports Hit@K, MRR, per-intent breakdown
✓ Plugs into your retriever — point --retriever at any callable (query, top, scope) → [results]

✗ Does not measure absolute quality — Hit@5=1.0 on a self-indexed corpus is optimistic by design
✗ Not a framework or hosted service — a harness you fork and customize
✗ Not a maintained product — personal tooling shared for the methodology

Comparison to RAGAS / DeepEval / Braintrust: Those tools gate answer quality (faithfulness, relevancy) via labeled golden sets or LLM judges. hitgate gates retrieval ranking label-free. Different layer, complementary — and the only one that requires zero labels to start. Full comparison: docs/COMPARISONS.md.

Quick Start (10 seconds)

pip install -e ".[hybrid]"   # harness + bundled hybrid retriever

# Index a repo and run the eval gate
RAG_SOURCE_ROOTS="$PWD" python -m ragcore.build
RAG_RERANK_AUTO=off python -m hitgate.run --label demo

# ...or point the gate at YOUR retriever
python -m hitgate.run --retriever myretriever:retrieve --label mine

What you get: Hit@5, Hit@1, MRR + per-case breakdown showing which queries failed and why.

CLI Commands

After pip install hitgate, the following commands are available on $PATH. Each also works as python -m <module> (e.g. python -m hitgate.run --label demo).

Indexing

rag-build — Build a searchable index from source code and docs.

RAG_SOURCE_ROOTS="/path/to/corpus" rag-build

• --incremental FILES — Reindex only specific files (e.g. --incremental src/core.py docs/api.md)
• --no-code — Skip source code; index docs only

rag-query QUERY — Search the indexed corpus.

rag-query "how does pagination work" --top 5 --format json

• --top K (default 5) — Return top K results
• --scope TYPE — Filter by source type (e.g. --scope code or --scope docs)
• --rerank {on|off|auto} (default auto) — Cross-encoder reranking
• --format {text|json} — Output format

Eval Gate

hitgate-run — Run the gate on a golden set and measure Hit@K, MRR.

hitgate-run --dataset eval/golden.jsonl --label baseline

• --top K (default 5) — Measure up to top K
• --label NAME (default baseline) — Label for output file
• --dataset PATH — Golden set (JSONL)
• --retriever MODULE:FUNC — Custom retriever (omit to use bundled hybrid)
• --rerank — Force cross-encoder reranking (bundled retriever only)
• --auto-rerank — Auto-trigger reranking on weak/ambiguous queries
• --verbose — Show per-case details

hitgate-init [ROOT] — Scaffold a label-free golden set from a corpus.

hitgate-init /path/to/corpus --output eval/golden.jsonl

• --output PATH (default eval/golden.jsonl) — Where to write the golden set
• --min-confidence {high|medium|low} (default medium) — Heuristic confidence threshold
• --limit N (default 0, unlimited) — Cap number of candidate cases

hitgate-demo [ROOT] — One-shot demo: build index, mine cases, run gate, print verdict.

hitgate-demo /path/to/corpus --limit 12

• --limit N (default 12) — Number of golden cases to mine

Analysis

hitgate-generate — Mine candidate golden cases from a corpus (auto-generated, needs curation).

hitgate-generate --output candidates.jsonl --min-confidence medium

• --output PATH (default hitgate/candidates.jsonl) — Where to write candidates
• --existing JSONL — Skip files already in an existing golden set
• --min-confidence {high|medium|low} (default medium) — Confidence threshold
• --llm — Use an LLM to paraphrase queries (requires OPENAI_API_KEY)
• --limit N — Cap total output cases

hitgate-diff BASELINE.json HEAD.json — Diff two eval result JSON files case-by-case.

hitgate-diff eval/baseline.json eval/ci.json

• --quiet — Summary line only, no per-case detail

hitgate-audit-contamination — Check golden set for data leakage (eval queries appear in corpus).

hitgate-audit-contamination --dataset eval/golden.jsonl

• --dataset PATH (default eval/golden.jsonl) — Golden set to audit

How It Works

Hybrid Retrieval

• Dense embeddings (intfloat/multilingual-e5-small) + BM25 lexical search
• Reciprocal Rank Fusion to combine both signals
• Code-aware tokenizer — splits camelCase/snake_case so "get user profile" matches getUserProfile
• Language-aware chunking — Python via AST, TS/JS/Shell via regex, fallback to word count
• Optional reranking — cross-encoder scoped to code queries only, graceful fallback if unavailable

The Eval Gate

1. Run your retriever against a golden set of queries (you write 20–99 JSON cases; this demo scores 99)
2. Measure Hit@1/3/5, MRR — does the expected file rank in top K?
3. Freeze a baseline when you like the results
4. Gate future changes — CI exits 1 if any metric regresses by >5pp

Config via env var (RAG_*); see ragcore/config.py.

Metrics Explained

Self-indexed results are optimistic by construction — point the gate at your own retriever for numbers that mean something.

Bring Your Own Retriever — 4-Step Quickstart

A retriever is any callable:

retrieve(query: str, top: int, scope: str | None) -> Sequence[Mapping]
# Results ranked best-first; each a Mapping with at least "path" (optionally "start_line")

0. Bootstrap candidate golden cases (optional)

RAG_SOURCE_ROOTS="/path/to/corpus" python -m hitgate.generate \
    --output hitgate/candidates.jsonl \
    --min-confidence medium

Review and curate to remove vague queries or wrong expected paths.

1. Write golden cases

{"query": "how does pagination work", "expect_path_contains": "api/pagination.py", "expect_scope": "code"}
{"query": "rate limit config", "expect_path_contains": "config/limits.yaml", "expect_scope": "code"}

Aim for 20–50 cases mixing identifier lookups and paraphrased queries. Save as any .jsonl.

2. Run your retriever

python -m hitgate.run \
    --retriever mypkg.myretriever:retrieve \
    --dataset   my_golden.jsonl \
    --label     baseline-v1

Outputs hitgate/baseline-v1.json with Hit@1/3/5/MRR + per-case breakdown.

3. Freeze the baseline

cp hitgate/baseline-v1.json hitgate/baseline.myproject.json

Edit _note to record conditions: corpus, model, date.

4. Gate future runs

BASELINE_FILE=hitgate/baseline.myproject.json \
python -m hitgate.run --retriever mypkg.myretriever:retrieve --dataset my_golden.jsonl --label ci

bash hitgate/check.sh hitgate/ci.json hitgate/baseline.myproject.json
# Exits 1 if any metric regresses >5pp

Drop-in CI: examples/retrieval-gate.yml

Diff two runs case-by-case

python -m hitgate.diff hitgate/baseline-v1.json hitgate/ci.json

Architecture & Performance

Self-Indexed Demo (Reproducible)

63 of 99 hit at rank 1; misses kept on purpose (no quiet dropping). Why Hit@5 is the gate: as the corpus grows, borderline cases drift rank. Hit@5 tolerates that noise; Hit@1 is for strict ranking. See CONTEXT.md (search "Drift over self-indexed corpus") for the fuller drift/self-indexed discussion.

External Corpus Benchmarks (Zero Tuning)

Finding: corpus module clarity predicts Hit@1 better than language or size. Clean functional boundaries → 0.85. Same-layer UI components → 0.59. Same tokenizer, zero corpus-specific tuning.

Selection bias: FastAPI is third-party; six of seven are the author's own repos. See docs/SWEEP.md for a broader 63-repo sample.

Full methodology and miss taxonomy: docs/METHODOLOGY.md

Use Cases

AI Assistant Memory

Gate retrieval quality in RAG-backed chatbots or assistants. Detect when corpus updates or embedding model swaps regress search. Deployed version of this exact use case: docs/two-channel-fastapi.md (auto-mined Hit@5=1.0 vs hand-labeled=0.92).

Want the retrieval engine + MCP server + agent-recall hooks, not just the eval gate? See shelfmark — same hybrid-retrieval lineage, packaged for agent memory instead of regression testing.

Document / Knowledge Base Search

Keep search quality stable as your docs grow. No user click data? Gate on retrievability. Ecosystem wrappers for LangChain / LlamaIndex: hitgate/adapters/README.md.

Code Search

Find code references and definitions. The hybrid retriever knows camelCase — useful for codebases with identifier-heavy queries. Same gate works for private codebases.

Extending

Core indexes code + docs + commits, nothing else. Tool-specific sources (assistant transcripts, code-graphs, other memory stores) plug in as optional adapters — hitgate/adapters/README.md.

What to Adopt (and What to Skip)

Adopt: The harness (hitgate/) — label-free regression gating and the --retriever interface. Plug in your own retriever.

Skip:

• Not a framework or hosted service (no plugin marketplace, no SaaS)
• Not state-of-the-art retrieval research (pragmatic, single-user, knows its ceiling)
• Not a maintained product (solo operator's personal tool; best-effort, no SLA)

Deliberately omitted: CONTRIBUTING, issue templates, badge walls. DECISIONS.md records what's left out on purpose and when it might reopen.

Next Steps

• Quick Start — run in 10 seconds
• METHODOLOGY.md — how the measurements work and their limits
• COMPARISONS.md — vs RAGAS, DeepEval, Braintrust, and hand-labeling
• ROADMAP.md — candidate experiments (no commitments)
• DECISIONS.md — what's out of scope and why
• CONTEXT.md — full vocabulary and decision history

License

MIT — see LICENSE.