codebase-rag 1.0.0

Symbol-aware, staleness-tracking RAG over a codebase, exposed over MCP; layers on rag-timetravel.

codebase-rag

A self-updating, symbol-aware vector index and reference graph of a codebase, exposed over MCP, so any MCP-compatible LLM agent can understand a whole repository cheaply instead of burning tokens on dozens of grep and glob calls.

Status: v0.1, working and tested. The PyPI name is not yet claimed.

Why

Agents working against a shared codebase pay for context in two bad ways: full-context stuffing (expensive, slow, breaks on large repos) or ad hoc grep/glob exploration (burns tool calls, produces inconsistent understanding across agents). Most of that grep traffic is not semantic search; it is structural traversal: "where is this defined", "who calls this", "what does this depend on", "what breaks if I change it". A vector index alone cannot answer those. codebase-rag pairs a semantic vector index with a symbol reference graph and serves both over MCP, so an agent answers those questions in one tool call instead of N greps, and the index stays in sync as the code changes.

What it gives an agent

Three layers, all queryable over MCP:

1. Vector layer: semantic search over symbol-level chunks (functions, methods, classes, module statements) for Python and JS/TS.
2. Graph layer: a reference/call graph (calls, imports, contains edges) with go-to-definition, find-callers, find-callees, dependencies, and neighborhood traversal.
3. Understanding layer: whole-codebase tools built on the graph: a token-budgeted repo map (PageRank over the reference graph), hybrid graph-aware retrieval, and change-impact analysis.

Architecture

codebase-rag/
  chunker/     tree-sitter symbol chunking (Python, TS/JS/JSX/TSX) + edge extraction + ignore rules
  store/       LanceDB tables (chunks, edges, annotations, repo_meta) + embedder fingerprinting
  graph/       name resolution + dependency-free PageRank
  sync/        git blob-hash staleness audit + pre-push hook backstop
  mcp/         the MCP tool surface (service core + server wiring)
  cli.py       init / reindex / status / serve / graph

Storage, embedding, and reranking are not reimplemented here. codebase-rag depends on rag-timetravel for the Embedder protocol, the reranker, and LanceDB dataset versioning. It uses a direct-write path: rag-timetravel's own ingest() re-chunks with a fixed-size splitter and has a hardcoded schema, so codebase-rag owns its own symbol-aware LanceDB tables and writes pre-chunked symbol rows directly, reusing only the embedder and reranker. This package is the code-aware layer on top: chunking, edges, staleness, sync, and the MCP tools.

Storage schema

Four LanceDB tables, versioned via rag-timetravel's dataset versioning:

• chunks: one row per symbol. Fields include chunk_id, file_path, symbol, symbol_kind, content, embedding, imports, git_blob_sha, start_line, end_line, repo_id, embedder_fingerprint.
• edges: one row per graph edge. Fields: id, src_symbol, dst_name, edge_kind (calls | imports | contains), file_path, git_blob_sha, start_line, repo_id.
• annotations: semantic notes attached to a symbol, versioned separately from chunks so reindexing a symbol never touches its annotations.
• repo_meta: per-repo metadata (repo_id, root_path, embedder_fingerprint, last_full_index_sha, languages).

Embedder fingerprinting

embedder_fingerprint is a hash of the embedder's model id and dimensionality, stored in repo_meta at init time. Every write path checks the incoming embedder's fingerprint before writing; on mismatch the write is rejected rather than silently corrupting search quality for a shared index. Switching embedders requires an explicit full re-embed.

Staleness detection

A chunk is stale when its stored git_blob_sha no longer matches the current blob hash of its file (git hash-object). status walks the index, recomputes current blob hashes, and reports ok / stale / deleted per file. This is a deterministic audit, independent of any wall-clock or version-count heuristic.

Sync

Two triggers keep the index fresh, both re-deriving chunks and edges from the file on disk (the index is never written directly by an LLM):

• Git pre-push hook (codebase-rag init --install-hook): the correctness backstop; catches edits from anything that did not go through an MCP client (IDE edits, merges).
• The reindex_file MCP tool: the latency optimization; an agent calls it right after editing a file so its own session sees fresh results without waiting for a push.

Concurrent reindex_file writes are safe: LanceDB uses manifest-based optimistic concurrency, and the store retries on commit conflict rather than overwriting.

Install

pip install codebase-rag        # once the PyPI name is claimed
# or, from source:
pip install -e .

Requires Python 3.10+ and a git repository as the source of truth for change detection. The default embedder (all-MiniLM-L6-v2, local, 384-dim) needs the optional local extra:

pip install -e ".[local]"

Quickstart

# 1. Index a repo and install the pre-push hook
codebase-rag init /path/to/repo --install-hook

# 2. Check freshness at any time
codebase-rag status

# 3. Reindex manually (full, or only what changed since a commit)
codebase-rag reindex --full
codebase-rag reindex --changed-since <sha>

# 4. Print the neighborhood of a symbol as text
codebase-rag graph UserService.authenticate --depth 1

# 5. Serve the MCP tools to an agent
codebase-rag serve

Point any MCP client at codebase-rag serve and the tools below become available.

Add it to an existing codebase

Five minutes, from the root of a git repository:

# 1. Install (with the local embedder extra so no API key is needed)
pip install -e ".[local]"        # or: pip install codebase-rag[local]

# 2. (optional) Tell it what to skip, on top of the built-in defaults
#    (node_modules, .venv, dist, build, __pycache__, *.min.js are always skipped)
cat > .codebaseragignore <<'EOF'
# gitignore syntax
migrations/
*.generated.ts
vendor/
EOF

# 3. Build the index and install the git pre-push hook
codebase-rag init . --install-hook

# 4. Confirm it indexed cleanly
codebase-rag status          # expect: ok=<N> stale=0 deleted=0

The first init embeds every symbol, so it takes a moment on a large repo; subsequent updates are incremental. From then on the index stays fresh two ways: the pre-push hook reindexes changed files on every push, and an agent can call reindex_file right after editing. Commit .codebaseragignore if you want it shared; the .codebase-rag/ index directory is local and should stay gitignored (add /.codebase-rag/ to your .gitignore).

For a team, everyone runs init once locally against the same checkout. The index itself is not committed; it is derived from source, so each clone rebuilds it deterministically.

Use it with a coding agent

codebase-rag serve is a standard stdio MCP server, so any MCP-capable agent can connect with the usual config. In every case the command is codebase-rag and the args are serve --repo <path>.

Claude Code (from the repo root):

claude mcp add codebase-rag -- codebase-rag serve --repo .

or add it to .mcp.json at the project root (shareable with the team):

{
  "mcpServers": {
    "codebase-rag": {
      "command": "codebase-rag",
      "args": ["serve", "--repo", "."]
    }
  }
}

Cursor (.cursor/mcp.json in the project, or the global ~/.cursor/mcp.json):

{
  "mcpServers": {
    "codebase-rag": {
      "command": "codebase-rag",
      "args": ["serve", "--repo", "/absolute/path/to/repo"]
    }
  }
}

Codex CLI (~/.codex/config.toml):

[mcp_servers.codebase-rag]
command = "codebase-rag"
args = ["serve", "--repo", "/absolute/path/to/repo"]

Any other MCP client (Windsurf, Zed, Continue, a custom Agent SDK host) uses the same command/args pair. If codebase-rag is installed in a virtualenv, point command at that env's executable (for example /path/to/.venv/bin/codebase-rag) so the agent launches the right one.

How an agent should use it

The tools are most effective in this order; a short system-prompt note like the following trains the agent to prefer them over grep:

This repo has a codebase-rag MCP server. Call get_repo_map first to orient. Use search_context for "how does X work", find_definition / get_callers / get_callees for navigation, and impact_of before changing a signature. Prefer these over grep. After editing a file, call reindex_file on it so later searches stay accurate.

A typical loop: get_repo_map to learn the important symbols, search_context("where are requests authenticated") to pull a function plus its call context in one shot, get_callers / impact_of to scope a change, edit, then reindex_file to keep the index in sync.

MCP tools

Vector and retrieval

• search_code(query, k=8, kind=None): semantic search over symbols, optionally filtered by symbol_kind (function | method | class | module_statement). Reranked before return.
• search_context(query, k=5, hops=1): hybrid graph-aware retrieval. Finds semantic entry points, then graph-expands each hops steps into one coherent, token-bounded subgraph. Nodes are tagged by role (match | caller | callee | dependency). Use this instead of a search followed by several follow-up greps.

Structural navigation (the grep replacements)

• find_definition(name): candidate definitions for a name (go-to-definition), resolved across files by name plus import hints.
• get_callers(symbol): every call site of a symbol.
• get_callees(symbol): everything a symbol calls.
• get_dependencies(file_path): the imports a file depends on.
• neighborhood(symbol, depth=1): the connected subgraph around a symbol, token-bounded.

Whole-codebase understanding

• get_repo_map(budget_symbols=50, file_path=None): a token-budgeted structural map. Ranks symbols by PageRank over the reference graph and returns the most central ones as a compact skeleton. Call this first to orient in an unfamiliar repo for a few hundred tokens instead of reading dozens of files. Pass file_path to focus the map on one file and its neighbors.
• impact_of(symbol, max_depth=3): reverse-reachability. The transitive set of symbols that depend on the given symbol. Check this before changing a signature.

Retrieval and bookkeeping

• get_symbol(symbol_id): a single chunk plus its attached annotations.
• get_file_context(file_path): all chunks for a file, ordered by line number.
• reindex_file(file_path): re-derive a single file's chunks and edges from disk. Call after any edit; skipping it leaves subsequent searches against that file stale.
• annotate(symbol_id, note, author="agent"): attach a semantic note to a symbol (never mutates derived chunks or embeddings).
• status(): the staleness audit (ok / stale / deleted per file).

Language support and resolution strategy

• Python (tree-sitter-python): top-level functions, classes (whole-class chunk plus one chunk per method), and a single module-statement chunk. Decorators attach to their target. Imports are collected once per file and attached to every chunk.
• JS/TS (tree-sitter-typescript): function declarations, named arrow/function-expression bindings, class methods, and exported const/type declarations. .tsx / .jsx components are chunked whole (JSX body included).

Graph edges are extracted from the same parse (no second pass): contains (class to method), imports (module to imported name), and calls (enclosing symbol to callee identifier). Resolution of a call/reference name to a concrete definition is intra-file precise, cross-file name-based: exact within a file, matched by name plus import hints across files. Full cross-file type resolution (the job of an LSP or type checker) is intentionally out of scope; unresolved names are still stored as useful candidate edges, and ambiguous names return multiple candidates.

Both chunkers skip generated files, node_modules, .venv, and anything matching a .codebaseragignore file (gitignore syntax).

Configuration

• Embedder: any model the rag-timetravel Embedder protocol accepts. Default all-MiniLM-L6-v2 (local). OpenAI-compatible and Ollama endpoints are supported via the rag-timetravel factory.
• Reranker: none by default; other rerankers from rag-timetravel can be selected.
• Default index location: <repo_root>/.codebase-rag/lancedb.

Development

# Tests (unit + integration are offline and fast; the slow e2e uses a real model)
pip install -e ".[test,local]"
python -m pytest -q                 # full suite
python -m pytest -q -m "not slow"   # skip the model-download e2e

Non-goals (v0.1)

• No non-git staleness backend; git is the source of truth for change detection.
• No languages beyond Python and JS/TS.
• No UI; CLI and MCP tools only.
• No direct LLM writes to chunk text or embeddings; the index is always re-derived from source.
• No hosted multi-tenant service; this ships as a library and CLI.

License

See LICENSE if present; otherwise unlicensed pending release.