repolens-search 0.11.0

Ranked, described hybrid (BM25 + semantic) search over a repo's docs + code + data (including gitignored content), plus a typed corpus linter. Built for repos worked by a grep-on-demand agent.

repolens

Ranked hybrid search + a typed hygiene linter for a repo's whole corpus. One local index over your docs, code purpose-lines, and database schema, so you can ask "where does X live?" in plain words and get the right few files back — ranked and described — instead of sifting a wall of grep hits.

Built for repos where an agent (e.g. Claude Code) works alongside a growing pile of markdown and greps on demand rather than keeping a semantic index. Hybrid search ships by default — pip install repolens-search gives you ranked BM25 fused with semantic embeddings out of the box (fastembed + sqlite-vec, both CPU, no service, no keys). It degrades to lexical-only (BM25) if the model can't load, and --lexical forces BM25 on demand. The first run downloads a ~200MB model once (cached durably); everything else is offline.

How it works

Install once, index per repo, then just search. Three steps — and the last one stays current on its own forever.

1. Install once, globally. pipx install repolens-search puts the repolens CLI on your PATH, hybrid search included. One time, ever. (The PyPI package is repolens-search — the name repolens was taken; the command you run is still repolens.)

2. repolens init once per repo. Run it in each repo you want indexed. It creates two things:

   • .repolens.toml — the config: what to index, ranking + semantic settings. Auto-discovers any SQLite DBs and wires their schema in.
   • .repolens/index.db — the search index itself, a disposable, gitignored cache that self-refreshes and can't drift. Delete it and it rebuilds; it's never committed.

3. repolens find "…" — self-refreshing. Every search re-indexes any changed files first, then searches, so results are never stale. You never manually rebuild.

That's the whole model: pipx install once → repolens init per repo → repolens find.

What it does

repolens find "…" — where does X live? Builds a local SQLite index and answers ranked queries with a one-line description and the passage that matched per hit — so you get the relevant text, not just a file path. Ranking is hybrid: BM25 (FTS5) for exact-term/identifier precision, fused with dense semantic similarity for paraphrase/meaning recall, combined by Reciprocal Rank Fusion (RRF). If the embedding model can't load it degrades to BM25-only (and says so once); --lexical forces BM25-only on demand.

• Markdown — full text.
• Code / config — indexed by each file's purpose line (from its docstring or leading comment), so repolens find "garmin ingest" returns scripts/ingest_garmin.py — "Pulls Garmin biometrics into the DB", not a wall of matches. The full module docstring is indexed too (BM25 + embedded), so a conceptual query finds the file even when your words aren't in its one-line purpose.
• Database tables (optional) — table + column names, so "where do trades live" resolves to a DB table.
• Frontmatter — any keys, schema-free. Every YAML frontmatter key is indexed into a sparse frontmatter(relpath, key, value) table, so docs with different conventions (paths:, name/description:, sector:) coexist in one repo — repolens imposes no schema and clobbers none. A total, dependency-free parser degrades nested/exotic YAML to searchable text.
• Respects .gitignore by default (in a git repo) — so secrets, .env, and anything you ignore stay out of the index. In a non-git directory a .gitignore can't be honored — repolens indexes everything there and prints a warning so you're never silently exposed. Opt into include_gitignored when you want ignored notes searchable (a personal-knowledge-repo mode).
• The index is a disposable, gitignored cache: it updates incrementally (only changed files, by content hash) and can't drift. repolens index --rebuild is the full backstop; delete it and it regenerates.

repolens lint — keep the knowledge base honest. Zero-LLM structural checks (dead links, empty files, malformed frontmatter, duplicate titles) and per-type field checks you declare in config (e.g. a doc in meetings/ must carry a **Date:**). A bundled pre-commit hook runs it and blocks a commit on errors — hygiene enforced, not hoped for.

repolens bench — prove hybrid earns its dependency. Scores hybrid vs lexical find on a committed gold set (benchmarks/acceptance.jsonl) and prints recall@k + MRR per query class, so the "semantic helps" claim is something you run, not something you trust. See below.

Does it help? (don't trust me — run repolens bench)

Rather than claim it helps, repolens ships a benchmark you run. repolens bench scores every query three ways against the same corpus — a literal grep baseline (ranked by distinct query terms matched, a fair reading of grep output), lexical find (BM25), and hybrid find (BM25 + semantic) — and reports recall@k, MRR, and a bootstrap 95% CI on the deltas. The gold set is small and self-authored (benchmarks/acceptance.jsonl: 18 query→gold-doc pairs on this repo's own files), so read the numbers as a directional signal on one corpus, not a general claim.

On this repo (bge-base, k=8):

arm	recall@8	MRR
grep (distinct-term) *	94%	0.52
lexical (BM25)	50%	0.34
hybrid	100%	0.67

Two honest takeaways, with the uncertainty attached:

• Hybrid clearly beats BM25 — ΔMRR +0.33, 95% CI [+0.17, +0.50] (excludes zero). Lexical is what you fall back to without the model, so this is the result that justifies shipping semantic on by default.
• Against a fair grep, hybrid wins on recall (100% vs 94%) but its MRR edge is within the noise — ΔMRR +0.16, 95% CI [−0.01, +0.33] (crosses zero at n=18). Grep is a stronger literal baseline than most tools admit; hybrid's real advantage is recall and the meaning-based queries grep can't reach, not a ranking blowout.

* The benchmark is charitable to grep — and that under-credits repolens. Real grep/rg doesn't rank: it returns matches in file order, unranked. The grep arm here is given a relevance ranking (by distinct query terms) that grep doesn't natively have, so the MRR comparison already flatters it. In actual use the gap is wider than the number shows: grep hands back an unranked pile of matches that you — or your agent — must read and judge (a real round-trip in time and tokens the benchmark charges grep nothing for), whereas repolens returns the files already ranked, each with the matching passage. That "ranked + passage vs sift-it-yourself" difference is a genuine repolens advantage the retrieval MRR doesn't measure — kept separate here on purpose, but real.

Run repolens bench on your own repo — that's the number that matters, not this one.

Who it's for

A repo that mixes prose/knowledge with code and is worked by an agent that greps on demand rather than maintaining a semantic index — Claude Code being the prime example. There, repolens gives a ranked, described answer across docs + code + data (and, when you opt in, your gitignored notes), plus lightweight enforced hygiene.

What it's not

Not a replacement for ripgrep (use rg for exhaustive literal/regex code search), not a full RAG system (it's the retrieval layer an agent does RAG with — it finds; the agent answers), and not a knowledge-management app. It's a hybrid findability + hygiene layer with one deliberate edge: it sees the whole corpus — prose, code docstrings + purpose-lines, and DB schema — and keeps it clean.

When to use repolens find vs rg

Grep when you know the string; repolens find when you know the concept but not the file.

• rg / grep → you know the literal string or regex, or you need every match. Fast, complete, literal.
• repolens find → "where does X live / which file handles Y" — you want the right few files, ranked and described, across docs + code docstrings/purpose-lines + DB schema (plus your gitignored notes, when you opt in).

Two things worth knowing about matching. A multi-word query is all-terms (every word must appear in the same file); if that returns nothing, repolens automatically broadens to any-term and tells you on stderr — so find "garmin deploy" still surfaces the closest files even when no single doc has both words. And matching is stemmed (ranking finds ranked) but does not split identifiers — search parse or frontmatter, not parseFrontmatter, to match a camelCase/snake_case name.

Install

pipx install repolens-search   # PyPI package name; the command it installs is `repolens`
# or: uv tool install repolens-search

Requires Python 3.11+. repolens depends on fastembed (ONNX embeddings, no PyTorch/CUDA/service), sqlite-vec, and numpy — all CPU-only, no service. (fastembed pulls onnxruntime, a prebuilt binary; on the rare platform where that won't install, repolens still runs lexical-only.)

First run has a one-time cost. The first index build embeds your whole corpus — fastembed downloads the model (BAAI/bge-base-en-v1.5, ~0.2 GB) once (cached under ~/.cache/repolens, override with REPOLENS_CACHE_DIR), then embeds every doc's chunks on CPU. Budget roughly a few seconds per document (a few hundred large markdown files can take several minutes). It's throttled by default ([semantic].threads = 2) so it won't max your machine — raise it (repolens index --threads 0 for all cores, or set [semantic].threads) for a faster one-off rebuild. After that it's incremental — only changed files re-embed, so day-to-day use is instant. Prefer no model? Set [semantic].enabled = false for lexical-only (BM25). On a build without loadable-extension support (sqlite3 is compiled without it on some platforms — notably stock macOS), repolens automatically falls back from sqlite-vec to a numpy brute-force vector search; it announces which path is active.

Heavy use? Point it at Ollama. With the default local fastembed, each repolens find runs as its own process and reloads the model every time — that model load is most of a hybrid query's latency. For an agent firing many queries, keep the model resident: set [semantic].provider = "http" and endpoint to a local Ollama (or any OpenAI-compatible /v1/embeddings) — it holds the model in memory across calls, so each find is just the embed + search. (Ollama's own core cap is server-side via OLLAMA_NUM_THREAD, not [semantic].threads.) fastembed stays the zero-config default; this is the upgrade for query-heavy workloads.

Quick start

cd your-repo
repolens init                 # writes .repolens.toml + .gitignore + a warm index + the pre-commit lint hook,
                              # and auto-discovers your DBs. --no-db opts out of discovery.
repolens index                # rebuild/update the index (incremental; a disposable cache)
repolens find "where's the deploy config"
repolens bench                # score hybrid vs lexical on the committed gold set (recall@k + MRR)
repolens lint                 # corpus hygiene + typed-record checks

Configure (.repolens.toml)

repolens init drops a commented starter. Declare your typed records by folder:

[types.meeting]
folder = "meetings"
recursive = true               # classify subfolders too
exclude = ["*draft*"]          # artifacts, not records
require = ["^\\*\\*Date:\\*\\*"]  # regex a conforming doc must contain (a warn if missing)

# Semantic (hybrid) search — ON by default. Set enabled = false for lexical-only.
# [semantic]
# model = "BAAI/bge-base-en-v1.5"   # short-passage retriever; fits the ~512 section-bounded chunks
# enabled = true
# chunk_tokens = 512                # per-chunk target; a chunk never crosses a heading
# threads = 2                       # throttle fastembed's CPU (0 = all cores)
# provider = "http"                 # OR bring your own: OpenAI-compatible /v1/embeddings
# endpoint = "http://localhost:11434/v1/embeddings"   # e.g. local Ollama (GPU) or a metered API
# api_key_env = "OPENAI_API_KEY"    # env var holding the key (never stored here)

# SQLite integration — index table/column NAMES (schema only, read-only).
# `repolens init` AUTO-DISCOVERS the DBs in your repo (including gitignored
# ones, skipping *.bak* backups) and fills this in. One or many:
# [integrations.sqlite]
# paths = ["data/app.db", "data/other.db"]   # legacy `path = "..."` also works

An explicit type: in a doc's YAML frontmatter overrides the folder rule.

SQLite auto-discovery. repolens init scans for *.db / *.sqlite / *.sqlite3 files — including gitignored ones (real DBs usually are), skipping backups and its own index cache — and writes the ones it finds into [integrations.sqlite] for you, so repolens find "where do trades live" resolves to a DB table with no hand-config. It reads only table and column names (via sqlite_master + PRAGMA table_info, opened read-only) — never row data. Pass repolens init --no-db to skip it, or edit the paths list by hand.

Use with an agent

repolens is a plain CLI, so any agent that can run a shell command can use it — nothing to wire up. Drop a short routing rule into the repo (a .claude/rules/ file for Claude Code, or AGENTS.md) so the agent reaches for it:

Default to repolens find "<what you're after>" to locate anything. Use rg / grep only for an exact string you need every match of, or a regex.

A ready-to-drop version is in docs/agent-rule-template.md. Every find re-indexes changed files first, so the agent's answers are never stale, and each hit comes back with the passage that matched — the agent gets the text, then reads the file with its own tools if it needs more. Because the index respects .gitignore, no ignored file contents leave the machine.

How the index stays correct

The index (.repolens/index.db, gitignored) is a cache derived from your files — never the source of truth. It updates incrementally: a files(relpath,size,mtime,hash) table stat-gates each file and re-indexes only those whose content hash changed (so a touch or a fresh clone re-hashes but doesn't re-index), reconciling deletes, all in one WAL transaction. WAL is what lets many agent sessions read (and the odd one refresh) the same index concurrently without locking each other out. The same content-hash drives embeddings (hybrid is on by default) — a changed file is re-chunked (section-bounded: chunks split on Markdown headings and never cross one, ~512 tokens each) and re-embedded, its vectors stored via sqlite-vec (or the numpy fallback); find fuses the BM25 and dense results with RRF, rolling per-chunk hits up to their parent document. A deleted file's chunks/vectors cascade away. repolens index --rebuild is the always-correct full backstop (and what CI runs); the index can't go stale, and anything uncertain just rebuilds.

Private logging (opt-in)

Set [log].enabled = true and repolens appends one JSON line per find and per embed to .repolens/events.jsonl — inside the gitignored cache dir, so the log is local and private (never committed, never leaves your machine). Off by default.

A find event records the query, mode, the matched file paths + scores, and timing; an embed event records the file, chunk count, model, and timing. It's metadata only — the matched passage text is never written to the log (relevant if you enable include_gitignored over private notes: the log captures which files matched and your query, not their contents). Writes never raise, so logging can't break a search. Beyond debugging retrieval, the find log accumulates the real queries you and your agents run against the repo — far better data for growing benchmarks/acceptance.jsonl than hand-written gold.

Roadmap

See ROADMAP.md. Next: a cross-encoder reranker over the fused top-N, then llms.txt export.

License

MIT © 2026 Justin Hawkes.