symgrep — code search that never lets you mistake not found for not there

A structural layer over ripgrep, built for AI agents. Same matching semantics as rg; a different output contract, and one extra question it can answer that no name search can.

The problem it exists for

An agent searches for something, gets zero hits, concludes it does not exist, and writes it again.

That is not carelessness — it is a correct reading of a wrong answer. rg, and symgrep's own expansions, match names. A function that does the job under a different name is invisible to every one of them. In one codebase, in one day, that produced:

share.scrub                  beside  share._scrub_text          same job, no shared word
adapters.call                beside  adapters.call_complete     the duplicate written an hour after a
                                                                tool was built to find this pattern
_save_state                  in three files, byte-identical
_load_state / _state_path    in four

So symgrep does two things about it.

Zero hits say what they do not mean. Every miss now ends with:

⚠ this is a NAME search. 0 hits means nothing is CALLED this — it does NOT mean nothing DOES this.

Free, no dependencies, and on its own it would have stopped several of the duplicates above.

And you can ask by meaning instead. --semantic takes a description of the job and returns the functions that already do it:

$ symgrep --semantic "make an LLM call and retry if the reply was cut off at the token limit" src/
  0.786  bulkgate.is_truncated    src/bulkgate.py:277
         Checks whether a generated response was truncated due to hitting token or length limits.
  0.688  gate._autotune           src/gate.py:1133
         Learns and applies a per-call token cap to prevent waste or premature truncation.
  0.683  adapters.call            src/adapters.py:172
         Runs a single prompt against a single model, enforcing per-call token budgeting.

A name search for that query returns 0. Those three are exactly what the person about to write a fourth implementation needed to see.

The output contract

1. Sites, not lines. Matches group by (file, enclosing symbol) with the symbol's span — no follow-up file reads just to learn context. Basis is labelled per symbol in trust order: ast (Python's parser) / treesitter (real grammars) / nearest_decl (regex declaration scan — LAST RESORT, weakest label) / module.
2. Mechanical ordering, never filtering. src > test > vendored, then match density. Nothing rg matched is dropped; truncation always reports counts and the top hidden files.
3. Honest zero-hits. A miss reports the scope searched, labelled mechanical expansions (case, snake/camel/Pascal, token co-occurrence) with per-variant file counts, and the NAME-search caveat above.
4. Unavailable is not empty. If --semantic has no index or no embedder it says so and exits non-zero. It never returns an empty result list, because that is indistinguishable from "nothing matches" and the whole point is to stop that confusion.

Retrieval, not judgement. symgrep never decides relevance. Cosine ranking proposes candidates the same way ripgrep proposes sites; both are handed over with their basis stated. The calling agent judges. Nothing is filtered by a threshold — a weak top score is reported as weak, which is a different statement from "nothing exists" and must not collapse into it.

Architecture

INTENT.md is the North Star — what symgrep is for, the invariants it commits to, and how each is enforced. docs/ARCHITECTURE.md is the "how" — the module map, the three-questions/one-contract model, the honesty invariant every module defends, and the non-goals (no index trusted blind, no dependency on a foreign indexer — symgrep owns its indexes and keeps them freshness-honest).

Install

Prerequisite — ripgrep (rg). symgrep IS ripgrep underneath: it shells out to rg for every name search. rg is a system binary, not a pip dependency, so install it separately:

brew install ripgrep          # macOS
apt install ripgrep           # Debian/Ubuntu
dnf install ripgrep           # Fedora
pacman -S ripgrep             # Arch
# other platforms: https://github.com/BurntSushi/ripgrep#installation

If rg is missing, symgrep fails with a clear, actionable error (never a silent zero-hit). An rg installed somewhere off PATH can be pointed to explicitly with SYMGREP_RG=/path/to/rg.

# Distribution name is `symgrep-codesearch` (the bare `symgrep` is parked on PyPI by the Semgrep team);
# the import name and console scripts are unchanged — `import symgrep`, `symgrep`, `symgrep-mcp`.
pip install symgrep-codesearch                 # `symgrep` + `symgrep-mcp` land on PATH
pip install 'symgrep-codesearch[semantic]'     # + local embeddings (veccore/fastembed) for --semantic
pip install 'symgrep-codesearch[mcp]'          # + fastmcp for the HTTP MCP transport
pip install -e /path/to/symgrep                # editable, for local development

Being importable matters more than it looks: while symgrep was a PYTHONPATH-only tool, a neighbouring repo that needed "every function with its scope-qualified name and span" wrote its own extractor rather than use enclosing.py, which already did it with an mtime cache. The duplicate was caused by the missing packaging.

Use

symgrep PATTERN [ROOT ...] [-i] [-F] [-U] [--glob G] [--max-sites N] [--within PAT2] [--usage] [--resolve] [--json]
symgrep --index src/                                   # build the semantic index (incremental)
symgrep --semantic "what the function would do" src/   # search by meaning

The index stores one sentence per function describing its job, keyed by body hash, so a rebuild only costs the functions that changed. Embeddings are local (fastembed / bge-small): no API key, no network, nothing about the code leaves the machine.

Writing those sentences needs a model, and symgrep holds no opinion about which — set SYMGREP_DESCRIBE_CMD to any command that reads a function body on stdin and prints one sentence, or install the claude CLI and it will use that. It will not fall back to summarising function names: names are what a name search already covers, and indexing them would rebuild the blindness this feature removes.

MCP (Claude Code)

claude mcp add symgrep -- /path/to/symgrep/bin/symgrep-mcp

One tool, two questions: search(pattern, roots) for names, search(pattern, roots, semantic=true) for jobs. Not two tools — the caller reaching for code search is exactly the caller who needs to ask by meaning, and a separate tool is one more thing to remember at the moment remembering fails.

Runs dependency-free (stdlib + ripgrep, with a stdlib JSON-RPC fallback when fastmcp is absent). Both transports are tested for parameter parity, because a flag live on one and missing from the other is the same half-wiring this tool exists to catch.

Tests

python tests/test_honest_recall.py

Pins the invariants that are easy to break quietly: a zero-hit result must carry the NAME-search caveat and a hit result must not, both MCP transports must expose the same parameters, and an unavailable semantic search must raise rather than return empty.

Reviewed

2026-08-19 — full 5-axis honestreview (per-file 5-vendor panel · concept · seam · invariant · name, plus silent/unwired), $0.42 API. Fixed from it: rg partial-coverage errors were swallowed when any match existed (now ⚠ COVERAGE INCOMPLETE); semantic.build pruned the WHOLE index on a partial-root build (now prunes only within its roots — 1,161 rows recovered from the sqlite freelist); non-UTF-8 paths crashed the search; describer truncated bodies to 4000 chars and one slow unit killed a whole index run; search/render name collisions across modules; duplicated rg command builder; import-time $SYMGREP_INDEX; JSON-RPC fallback answered parse/unknown-method with silence. Second pass (all remaining high/medium): rg stderr pipe could deadlock on large --no-ignore trees (now drained concurrently); semantic index key was <stem>.<name> so same-named files overwrote each other (now <realpath>::<symbol>, schema v2 with in-place migration); --semantic ignored roots and a changed embed model returned [] silently (both now honest); stale rows survived an undescribable rewrite; symgrep --index src indexed the cwd; py3.9-incompatible annotation would stop the MCP server; unbounded symbol cache; files_searched reported files-with-matches. Regression tests: tests/test_review_regressions.py. Open (design, not bugs): path_class is a directory-name convention used for ORDERING only and now prints its basis; the agentic source for "what IS this file" is a catalog (warden), not symgrep.

2026-08-29 — deep wave (all 32 files × 5 vendors, ~$0.5 billed / rest plan-served) + invariant (intent) + silent. Fixed the confirmed bugs: a malformed bulk-result JSONL line aborted a chunk (now counted + skipped, batch still lands); the freshness worker silently DROPPED a batch on a failed/timed-out --index (now re-queued, retried); .class constant-pool strings decoded as plain UTF-8 not JVM modified-UTF-8 (now a proper mUTF-8 decoder); a non-numeric schema_version and a bad SYMGREP_MAX_PARSE_BYTES could crash (now guarded); NotebookEdit events used the wrong payload key; the JSON-RPC fallback leaked a traceback; test-honesty (rglob for subpackages, a vacuous system-file assertion, env restore). The invariant axis flagged the meaning-by-regex rule — adjudicated a FALSE POSITIVE (as before): _cos/_rank are retrieval handed to the caller, judge_is_code_search already calls the LLM (regex is only the prefilter), _identifier_tokens/_is_generated are format-parsing; the one actual meaning decision (the description) IS an LLM. So the intent holds. name/concept axes hit honestreview-side tracebacks (their bug).

Adoption (measured) and the nudge hook

Transcripts 2026-08-02 → 08-19 (MCP registered, description says "use instead of Grep"): 2% of code-search calls used symgrep (54 / 2,853). The built-in Grep tool was used once — agents search with rg/grep in Bash. Description-steering does not change behaviour; a hook does.

hooks/symgrep_nudge.py (PreToolUse, matcher Bash): prefilter by POSITION (rg/grep first in a pipeline segment — ps | grep is never touched), then a MODEL judges "search vs utility" (haiku via spendguard, plan lane, refuse-metered, purpose=symgrep-nudge, verdicts cached). Code search → blocked with the symgrep command; utility (--files, --version) → allowed; any failure → allowed (fail-open). Escape hatch: SYMGREP_ALLOW_RG=1 rg …. Register in ~/.claude/settings.json under hooks.PreToolUse.

Freshness (self-maintaining index)

Three layers keep --semantic honest about time, measured against the failure mode "silently ranking over old descriptions":

1. PostToolUse hook (hooks/symgrep_index_freshness.py): every Write/Edit of an indexable file inside the index's own recorded scope enqueues it; a detached worker drains the queue in debounced batches through symgrep --index (content-hash cached, spendguard-routed, refuse-metered). Hook latency ~50ms; fail-open everywhere.
2. Query-time staleness signal: every indexed row carries the file's (mtime_ns:size) stamp from when it was last VERIFIED (described or hash-checked). --semantic output states how many in-scope files changed on disk since — a stat-stamp SIGNAL, explicitly not a content verdict (content truth is body_hash, re-checked by every build; a touch makes the signal conservatively noisy, never falsely fresh together with size).
3. Weekly mop-up (launchd com.ashdamle.symgrep-index-refresh, Sun 04:10): incremental re-index of the configured roots — catches out-of-band changes (git pull, other machines) the hook cannot see.

Backup: launchd com.ashdamle.symgrep-index-backup (03:55) ships a consistent sqlite snapshot to B2 (rolling latest + Monday weeklies).

Bulk index builds ($0, concurrent across plan lanes)

When spendguard is present, symgrep --index fans the describe work for large todo lists (≥ $SYMGREP_BULK_MIN_TODO, default 10) across the idle subscription lanes concurrently via spendguard lanes --bulk symgrep-index --jsonl --system-file … --refuse-billed --checkpoint … — $0 by construction (a lane miss is an error row → that unit stays UNDESCRIBED and re-runnable, never a bill), content-keyed resume on crash, and per-row provenance (described_by = "<model> via <lane>") stored in the index. The fan is confined to completion lanes (SYMGREP_DESCRIBE_LANES, default zai-coding,codex,gemini): a conversational agent lane treats the function body as a chat turn and returns junk, so a describer must describe, not converse (tests/test_describe_lane_confinement.py; measured 2026-09-10 over 23,332 rows: the claude-code lane returned 80–87% conversational non-answers before it was excluded). Small batches (the freshness worker's 1–3 files) stay per-unit. Disable with SYMGREP_BULK=0; --reindex forces re-describe after a describer/instruction change.

Macro-bootstrap of a large STATIC tree (OpenAI Batch API). Plan lanes are right for incremental freshness but slow and quota-bound for tens of thousands of functions in a legacy tree that is not being edited. tools/batch_describe.py is the other tradeoff: build emits provider-neutral TASKS ({custom_id, content}, the function body as content) + a shared system file + a custom_id→metadata sidecar; spendguard builds the per-model batch envelope and runs the gated, estimate-first submit/fetch (~50% off, ≤24h turnaround, never touches plan quota); ingest writes the descriptions non-clobbering (a key that already has a description is skipped and counted, never overwritten) and LOUD (malformed / unknown-id / errored / empty rows each counted, every missing id named for re-submit), then embeds locally. symgrep never hand-rolls the request envelope — models.py in spendguard owns tokens-param / reasoning / output floor. This is deliberately NOT for actively-edited code (a 24h turnaround would make just-written code unsearchable until tomorrow); the freshness hook owns that. tests/test_bulk_describe.py.

Index scope = ALL of ~/Documents/claude (as of 2026-08-25): one root, so the freshness hook and weekly refresh inherit full-estate coverage from indexed_roots automatically. Scope rules are the search's own: hidden dirs, vendored/generated trees, and minified files are excluded; iCloud-dataless placeholders are skipped (reading one hangs) and re-enter once materialized.

Structural search (--structural) — a third question, native, no new dependency

Beside name (regex) and meaning (--semantic), symgrep matches by AST shape — pattern-by-example over the tree-sitter grammars it already loads (leverages ast-grep's idea; takes no ast-grep binary):

symgrep --structural '$X.get($K, $D)' src/     # dict.get with a default, however X/K/D are spelled
symgrep --structural 'foo($$$)' src/           # any call to foo, any arguments
symgrep --structural '$X == $X' src/           # a value compared to itself (repeated metavar = same text)

$NAME matches one AST node (a repeated $NAME must match the SAME text); $$$ is variadic. Matches flow through the normal contract — grouped by enclosing symbol, ordered, basis=ast-pattern. Scope is symgrep's own file walk, identical to name search. A language with no grammar, or a pattern that won't parse, is reported unavailable/skipped (mechanical status), never silently text-matched. On the MCP: search(pattern, roots, structural=true[, lang]), structured output carries the flat hit shape.

Refining a name search: --within and def/reference anchors

Two refinements make a name search answer a sharper question without leaving the ripgrep base.

--within PAT2 keeps only sites whose enclosing symbol also contains a match of PAT2 — the "inside any function whose body also mentions X" query, as a same-symbol co-occurrence:

symgrep 'subprocess' src/ --within 'json'      # functions that use BOTH subprocess and json

Honesty holds: a site whose symbol span is unknown (basis nearest_decl/module — no reliable [start,end]) cannot be tested for containment, so it is dropped and the drop is counted in the coverage line, never silently kept or discarded.

Definition vs reference anchor. On a name search, each occurrence is tagged: an occurrence on its enclosing symbol's declaration line, where that symbol's own name matches the query, is a definition (marked ▸def in the rendered output); every other occurrence is a reference. This is the Kythe/OpenGrok distinction, computed from the AST the sites already carry — no index, no name-guessing. A query that names no declared symbol yields zero definition anchors (all references), and an import or whole-file (basis=module) occurrence is never miscalled a definition. On the MCP, each name hit carries def_lines (the declaration-line subset of match_lines) and role: definition|reference.

Usage / possibly-unused (--usage). Summarises a name into definitions vs references and flags the defined-but-never-referenced case:

symgrep 'build_report' src/ --usage       # definitions: 1 · references: 7 · verdict: DEFINED and REFERENCED
symgrep 'some_helper' src/ --usage         # verdict: DEFINED but 0 references — POSSIBLY UNUSED

verdict ∈ defined_referenced | defined_unused | undefined_referenced (used here, defined elsewhere) | absent. It counts all occurrences (never the display cap — a truncated count could make a used symbol look unused). It is LEXICAL, by name: a same-named symbol elsewhere, dynamic dispatch, string-built calls, or use outside these roots are invisible to it — so defined_unused is a hint to confirm, never proof. On the MCP: search(pattern, roots, usage=true) returns {verdict, n_def, n_ref, definitions, references, lexical:true}.

Cross-file resolution (--resolve, R0). A step past lexical --usage: it builds a cross-file symbol table from the definitions symgrep extracts, and ties references to the actual definition(s) they could bind to — surfacing ambiguity instead of guessing.

symgrep 'build_report' src/ --resolve     # definitions: 1 · verdict: one definition in scope
symgrep 'main' src/ --resolve              # definitions: 2 — AMBIGUOUS: candidates among both, not one winner

verdict ∈ resolved_unique | ambiguous (>1 definition of the name) | external (referenced, defined elsewhere) | absent. Where two files define foo, R0 lists both and says AMBIGUOUS rather than picking one.

R1 narrows each reference to a single binding where Python import/scope makes it certain — so an ambiguous name still resolves per use-site:

symgrep 'estimate' src/ --resolve
  definitions: 3 — AMBIGUOUS …
  references: 40 — R1 bound 31 to a single definition (import/scope), 9 left as candidates:
    a/user.py:12  →  a/core.py:4  [resolved-import]  via `from a.core import estimate`
    a/core.py:9   →  a/core.py:4  [resolved-local]

Each reference carries its own basis: resolved-local (defined in this file), resolved-import (from m import foo → the definition in m), resolved-external (imported from outside the scanned roots), resolved-attr (R2/R2b: an attribute access whose receiver type is certain — self./this.method → the enclosing class's member; Class.member; mod.member for an aliased module import; or x.member where x was bound once, unconditionally to ClassName()/new Ctor() in scope — R2b local type tracking), or candidate-name when it declines — a star/default import, or an attribute on an instance of unknown type: a factory call, a parameter, or a reassigned / conditionally-assigned variable stays at R0's candidates, never a guessed winner. R1 covers Python (stdlib ast) and JavaScript/TypeScript (.js/.jsx/.mjs/.cjs/.ts/.tsx, ES-module + CommonJS require, via tree-sitter); a language with no resolver stays at R0. Resolvers plug into a registry (resolve_<lang>.py), so adding one (Go next) touches no shared code. Recomputed live; the persistent delta-graded index is R's later refinement (see docs/IMPROVEMENT_PLAN.md). On the MCP: search(pattern, roots, resolve=true) returns n_refs_bound / n_refs_candidate and per-reference bases.

HTTP MCP (for programmatic consumers)

symgrep-mcp --http serves the same search tool at POST http://127.0.0.1:4911/mcp (env: SYMGREP_MCP_HOST/PORT) — stateless JSON-RPC (json_response, no session handshake), loopback-only. Built for engines like honestreview that need symgrep WITHOUT spawning a stdio server per call or importing in-process into a venv that lacks fastembed.

Estate loopback port registry (one engine per port; a collision means consumers silently query the wrong engine): 4900 ccwatch · 4910 7thsense · 4911 symgrep. The SessionStart ensure-hook probes serverInfo.name on its port and ANNOUNCES a collision instead of assuming a live port is ours.

The exact contract (verified 2026-08-25, ~15ms/call):

• No handshake needed: each request is independent. initialize is optional and answers serverInfo: {name: "symgrep", version: <package version>} — the identity probe.

• Headers: Content-Type: application/json and Accept: application/json, text/event-stream (fastmcp requires both media types in Accept even though the reply is plain JSON).

• Call: {"jsonrpc":"2.0","id":1,"method":"tools/call","params":{"name":"search","arguments":{…}}} with the same arguments as the stdio tool: pattern, roots (required), case_insensitive, fixed_string, multiline, globs, include_ignored, max_sites, semantic, k.

• Reply: result.content[0].text = the same rendered report the stdio tool returns (honest zero-hits, FRESHNESS line, scope line included). Tool errors arrive as JSON-RPC errors, not empty results.

• Structured output: pass "structured": true (or the alias "json_output": true) and content[0].text is JSON with a mechanical top-level status and ONE flat hit shape for both search modes — no prose parsing, ever:

  • {status: "ok", mode: "name"|"semantic", hits: [{symbol, path, line, start_line, end_line, kind, basis, score?, desc?, match_lines?}], …} — name hits carry match_lines; semantic hits carry score + desc and resolve the span from the file's AST. scope (semantic) carries the staleness/coverage counters.
  • {status: "no_index", …} when the semantic index isn't built (fixable by symgrep --index) vs {status: "unavailable", reason: …, message: …} when it cannot run (no embedder / corrupt). These are distinguishable without reading the prose — the honesty feature a program needs. Languages. Source is handled by real parsers: Python (ast), and tree-sitter grammars for JS/TS, Java, Go, Rust, C/C++, Ruby, PHP, C#, Kotlin, Scala, bash — name-search + semantic, basis=ast/ treesitter. Compiled .class bytecode is the one path that is not source: ripgrep cannot search binary, so a JVM classfile's constant pool is read directly (pure Python, no JDK) to list its class, methods (overloads disambiguated by descriptor), and fields with basis=classfile — line spans only when the class carries debug info, honestly absent otherwise. It feeds the structural file_symbols tool (a dependency jar's symbols become queryable without source), and is deliberately kept OUT of the semantic index: a one-sentence "what it does" from bytecode would be a confident-wrong-answer.

• Two more tools (same structured discipline, both transports): file_symbols(path) → {status, basis, symbols: [{name, kind, start_line, end_line, basis}]} (the AST layer without an in-process import — for def-span lookups); indexed_roots() → {status, roots: […], index_path} (so an un-hit query reads as "not indexed here" vs "indexed, nothing near").

def mcp_query(engine_url, tool, args, timeout=60):
    body = json.dumps({"jsonrpc": "2.0", "id": 1, "method": "tools/call",
                       "params": {"name": tool, "arguments": args}}).encode()
    req = urllib.request.Request(engine_url, data=body, headers={
        "Content-Type": "application/json", "Accept": "application/json, text/event-stream"})
    with urllib.request.urlopen(req, timeout=timeout) as resp:
        return json.loads(resp.read())

mcp_query("http://127.0.0.1:4911/mcp", "search",
          {"pattern": "record a charge to the ledger", "roots": ["~/Documents/claude/llm-spendguard"],
           "semantic": True, "k": 5})

Supervision is SESSION-CONTEXT, not launchd: macOS TCC denies launchd agents ~/Documents (measured: the launchd weekly-refresh died Operation not permitted/exit 126 on every run), so a SessionStart hook (hooks/symgrep_http_ensure.py, containment-asserted spawns, fail-open) revives the server — and the weekly mop-up refresh, stamped only on success — with the session's inherited file access.