RepoDigest

A local-first codebase context synthesizer and token budgeter.

RepoDigest statically analyzes a Python codebase — no network calls, no external services, no code ever leaves your machine — and turns it into a token-budgeted, LLM-ready context around a target symbol, file, or plain-English question. It replaces "paste the whole repo into the prompt" with a deterministic pipeline: parse the AST, build the call graph, rank or target a symbol, and pack outward from it until the budget runs out.

The test badge reflects the local pytest suite as of the latest commit — there is no CI workflow configured yet, so treat it as a snapshot, not a live status check.

---

Table of Contents

• The Core Problem & Motivation
• System Architecture & Algorithms
  • 1. AST Parsing Pipeline
  • 2. Dependency Graph Construction
  • 3. Context Packing Algorithm
  • 4. Local BM25 Ranking Engine
• Installation
• Complete CLI Reference & Workflows
• Programmatic Python API
• Test Verification & Benchmarks
• Development & Solo Authorship

---

The Core Problem & Motivation

Feeding an LLM "the whole codebase" is the default move when a model needs context it doesn't already have — and it's a bad one, for three compounding reasons:

1. Context window bloat and cost. Every additional token in the prompt is billed, whether or not the model needs it. A 50-file service dumped wholesale into a prompt might be 200K tokens of which perhaps 2K are actually relevant to the question being asked. That's not a rounding error — it's two orders of magnitude of wasted spend, repeated on every single call.

2. "Lost in the middle." Even when a model's context window is technically large enough, published attention studies (e.g. Liu et al., "Lost in the Middle: How Language Models Use Long Contexts") consistently show that model recall of information degrades — often severely — for content placed in the middle of a long context, versus the beginning or end. Dumping an entire repository doesn't just cost tokens; it actively degrades the model's ability to find and use the one function that actually matters, by burying it in surrounding noise.

3. Irrelevant surface area invites wrong answers. More unrelated code in context means more opportunities for the model to anchor on a similarly-named function in an unrelated module, or to "helpfully" incorporate a pattern from code that was never meant to be relevant to the task.

Why not just concatenate files?

Naive bundling scripts (cat all .py files into one blob, optionally with file-path headers) are simple, but they have no notion of relevance or relationship. They can't answer "what depends on this function" or "what does this function depend on" — every file is either fully in or fully out. They also can't do interface-only rendering: a caller either gets the whole 200-line implementation or nothing, when often it only needs the four-line signature to reason correctly about how to call it.

Why not just use vector-only RAG?

Embedding-based retrieval (chunk the repo, embed the chunks, retrieve by cosine similarity) solves part of the relevance problem, but introduces its own failure modes:

• Chunk boundaries don't respect code structure. A fixed-size or paragraph-based chunker will routinely split a function in half, or merge the tail of one function with the head of the next — handing the model a syntactically incomplete fragment.
• No structural guarantees. Semantic similarity between a query and a docstring says nothing about whether that symbol is actually reachable from — or a dependency of — the code the user is working on. A vector store has no concept of "caller" or "callee."
• Non-determinism and drift. Embedding models change between versions, similarity thresholds need tuning, and the same query can silently retrieve different results after a model upgrade.
• Infrastructure weight. A vector database is one more moving part, one more thing to index, invalidate, and keep in sync with the source tree, for a problem that — for a single codebase on a single machine — a deterministic call graph solves for free.

RepoDigest takes a deterministic AST + DAG approach instead: parse the exact source with Python's own compiler front-end, build an exact (name-resolved) call graph from it, and traverse that graph outward from a precise starting point. The same target, the same repository state, always produces the same packed context — no embeddings, no similarity thresholds, no drift.

---

System Architecture & Algorithms

flowchart LR
    subgraph Input
        A[Python source tree]
    end

    A --> P["parsers/py_parser.py\nAST symbol extraction"]
    P --> G["graph/call_graph.py\nCross-file call graph"]
    P --> B["packer/budget.py\nSymbol registry"]
    G --> B
    Q[["Query or --entry"]] --> R["search/ranker.py\nBM25 ranking"]
    R -->|seed symbol| B
    B --> S["packer/serializer.py\nXML / Markdown"]
    S --> O[Token-budgeted context]

1. AST Parsing Pipeline

repodigest.parsers.py_parser uses Python's built-in ast module — the same parser CPython uses to compile your code — so extraction is exact, not regex-approximated. parse_source/parse_file walk a module's top-level body and, for every ClassDef, its nested body, extracting:

source text
     │
     ▼
 ast.parse()                     # exact syntax tree, same as the interpreter uses
     │
     ▼
 walk tree.body (top level only) ─────────────┐
     │                                        │
     ├─ Import / ImportFrom  → ImportInfo      │  (module, name, alias, lineno)
     ├─ FunctionDef/AsyncFunctionDef → FunctionInfo
     │     ├─ args (positional/varargs/kwonly/kwargs, with annotations & defaults)
     │     ├─ returns (unparsed annotation)
     │     ├─ docstring (ast.get_docstring)
     │     └─ source: full body OR signature-only stub
     └─ ClassDef → ClassInfo
           ├─ bases / decorators
           ├─ docstring
           ├─ methods: tuple[FunctionInfo, ...]  (same extraction, is_method=True)
           └─ source: full class body OR every method stubbed

Every function, method, and class carries two renderable forms, selected by mode:

• mode="full" — the verbatim original text, decorators included. (This is not simply ast.get_source_segment(source, node): that function anchors on node.lineno, which for a decorated function/class points at the def/class line rather than the decorator above it — silently dropping decorators. RepoDigest's _full_source helper slices from the first decorator's line instead, when one is present.)
• mode="signature" — the body is rebuilt as an AST stub (_render_signature / _render_class_signature) keeping only the docstring and an ... placeholder, then re-serialized with ast.unparse. For a class, every method is stubbed this way, so the whole class's public shape is visible in a fraction of the tokens its implementation would cost.

This is the mechanism behind the CLI's --signatures-only flag: it doesn't strip lines with a regex — it round-trips through the AST, so decorators, type annotations, and multi-line default arguments are always rendered correctly.

2. Dependency Graph Construction

repodigest.graph.call_graph.CallGraph.from_directory(root) builds a fully-qualified, cross-file symbol graph in two passes:

Pass 1 — Symbol table. Every file under root is parsed, and every function, method, and class is assigned a fully-qualified name derived from its file path and nesting:

src/repodigest/packer/budget.py  →  module qualname "packer.budget"
    class ContextPacker          →  "packer.budget.ContextPacker"
        def pack(...)            →  "packer.budget.ContextPacker.pack"
    def build_symbol_registry    →  "packer.budget.build_symbol_registry"

These qualnames are collected into a symbol_table: dict[str, set[str]] mapping simple name ("pack") → the set of fully-qualified names that share it (there can be more than one pack across a codebase).

Pass 2 — Edge resolution. Every function/method body is walked for ast.Call nodes. For a call like helper() or self.other(), the callee's simple name (helper, other) is looked up in the symbol table, and a directed edge caller_qualname -> callee_qualname is added for every qualified symbol sharing that name.

This is a deliberate name-based resolution trade-off, not full import-alias tracing: it never needs to resolve import numpy as np chains or re-exports to find a match, which keeps it fast and dependency-free, at the cost of occasionally over-linking two unrelated functions that happen to share a name (e.g. two different run() methods on unrelated classes). For the purpose this graph serves — finding plausibly relevant neighboring context under a token budget, not proving program correctness — that trade-off is the right one.

Edges are stored bidirectionally for O(1) lookups in both directions:

self._callees: dict[str, set[str]]   # caller -> {callees}
self._callers: dict[str, set[str]]   # callee -> {callers}

graph.get_callees("packer.budget.ContextPacker.pack")   # downstream: what `pack` calls
graph.get_callers("packer.budget.ContextPacker.pack")   # upstream: what calls `pack`

3. Context Packing Algorithm

ContextPacker.pack(target) is the formal core of RepoDigest. Given a target (an explicit symbol, a file path, or a BM25-ranked seed), it runs as follows:

1.  roots ← resolve(target)
        - if target is a known symbol:      roots = [target]
        - if target is a file path:         roots = every top-level function/class
                                             defined directly in that file
                                             (nested methods excluded — they're
                                             already inlined in their class's
                                             full source)

2.  order ← roots ++ bfs_alternate(roots)

    bfs_alternate(roots):
        seen ← roots
        frontier_down, frontier_up ← roots, roots
        loop while either frontier is non-empty:
            # one hop of DOWNSTREAM (callees) for every node in frontier_down
            next_down ← unseen callees of frontier_down, sorted, marked seen
            # one hop of UPSTREAM (callers) for every node in frontier_up
            next_up   ← unseen callers of frontier_up,   sorted, marked seen
            append next_down, then next_up, to order
            frontier_down, frontier_up ← next_down, next_up

3.  used ← 0 ;  packed ← [] ;  skipped ← []
    for symbol in order:
        text   ← full_text   if symbol ∈ roots or signatures_only == False
                  else        signature_text
        tokens ← count_tokens(text)
        if used + tokens > budget:
            skipped.append(symbol)      # SKIP — never truncate — keep scanning
            continue
        packed.append((symbol, text, tokens))
        used += tokens

4.  return PackedContext(packed, skipped, total_tokens=used, budget)

Three properties fall directly out of this design:

• Distance-ordered relevance. Because expansion alternates one hop of downstream, one hop of upstream, per round — rather than exhausting one direction first — a symbol two hops downstream and a symbol two hops upstream are considered at roughly the same priority, instead of one direction systematically starving the other.
• The budget is never exceeded. The check is used + tokens > budget, evaluated before a symbol is ever appended — there is no code path that produces total_tokens > budget.
• The "skip-on-overflow" heuristic. When a candidate doesn't fit, RepoDigest does not truncate it (a truncated function body is often worse than useless — syntactically broken, semantically misleading) and does not stop the whole traversal (a heuristic like "stop at first overflow" would leave remaining budget on the table if a smaller, more-distant symbol would still fit). It skips that one candidate and keeps scanning, so the final packed set is the best-effort greedy fit of closest-first symbols into the remaining space.

4. Local BM25 Ranking Engine

When you don't know the exact symbol name, repodigest.search.ranker.SymbolRanker scores every symbol against a free-text query with a from-scratch BM25 implementation — no external search library, no vector store.

Sub-token tokenization. Symbol and identifier names are rarely single dictionary words — compute_loss and computeLossValue both need to match a query containing "loss." The tokenizer handles both naming conventions:

_CAMEL_BOUNDARY = re.compile(r"(?<=[a-z0-9])(?=[A-Z])")   # insert a boundary before a capital
_NON_ALNUM = re.compile(r"[^A-Za-z0-9]+")                 # split on everything else, incl. "_"

tokenize("compute_loss")      # -> ["compute", "loss"]
tokenize("computeLossValue")  # -> ["compute", "loss", "value"]

BM25 scoring. For a query term with document frequency df across N documents (symbols), against a document of length dl and average document length avgdl:

idf(term)        = ln( (N - df + 0.5) / (df + 0.5) + 1 )

score(q, doc)     = Σ_{term ∈ q}  idf(term) · tf(term, doc) · (k1 + 1)
                                  ─────────────────────────────────────
                                  tf(term, doc) + k1 · (1 - b + b · dl/avgdl)

with the standard tuning constants k1 = 1.5 and b = 0.75 — k1 controls how quickly additional term occurrences saturate in their contribution to the score, and b controls how strongly a document's length is normalized against the corpus average (b=0 disables length normalization entirely; b=1 fully normalizes).

Corpus construction. Each symbol's document is its qualified name + docstring + rendered signature — i.e. its signature_text (which already embeds the docstring, per §1) prefixed with its qualname — deliberately not its full implementation body. This keeps a query like "where is the loss calculated" matching on the vocabulary a human would use to describe a symbol's purpose, rather than being drowned out by common keywords (self, return, if) that appear inside unrelated function bodies far more often than in any docstring.

Automatic root seeding. When --entry is omitted and --query is given, the CLI builds this corpus over every symbol in the registry, ranks it, and feeds the single top-scoring symbol (SymbolRanker.top(query)) straight into ContextPacker.pack(...) as the root — the query never touches the packer directly, it only ever selects where the deterministic graph traversal begins.

---

Installation

Requires Python 3.10+. This project uses a local venv/ checked in at the repo root.

# from the repo root
./venv/bin/pip install -e .

This installs RepoDigest in editable mode and registers the repodigest console script (wired via [project.scripts] in pyproject.toml) inside venv/bin/.

---

Complete CLI Reference & Workflows

After installing, use the repodigest binary directly (./venv/bin/repodigest, or just repodigest once venv/bin is on your PATH).

repodigest pack <path>

Pack a token-budgeted context around a target symbol, file, or natural-language query.

Flag	Type	Default	Description
PATH	argument	—	Directory to scan (the repo/package root)
--budget	int	4000	Token budget for the packed context
--entry	str	None	Target symbol (pkg.mod.Class.method) or file path, relative to PATH
--query	str	None	Natural-language query; the top BM25 match seeds the pack
--format	xml | markdown	markdown	Output serialization
--signatures-only	flag	off	Render non-root dependencies as signatures only (root is always full)
--output	path	stdout	Write to a file instead of printing

Exactly one of --entry or --query must be given; passing neither raises a usage error, and an unmatched --query (BM25 score of zero against every symbol) raises a clear CLI error rather than silently packing nothing.

Example — target a known symbol, Markdown to stdout (tight budget, to show skip-on-overflow):

$ repodigest pack src/repodigest --entry packer.budget.ContextPacker.pack --budget 500

Real captured output (some fenced code blocks omitted below with # ... for brevity — the actual output includes the full body of every packed symbol):

## src/repodigest/packer/budget.py

def pack(self, target: str) -> PackedContext:
    ...
# ... (4 more fully-packed symbols from budget.py and tokenizer.py) ...

---
**Token usage:** 499 / 500 (5 symbols packed, 6 skipped)

The loop only ever adds a symbol if it fits, so total_tokens never exceeds budget — here it comes right up against the ceiling without going over, with the 6 next-closest candidates skipped rather than truncated.

Example — find the relevant code by asking a question instead of naming a symbol:

$ repodigest pack src/repodigest --query "Where is the token budget enforced?" --budget 4000

Example — keep the target's full body, but dependencies as signatures only, to fit more breadth into the same budget:

$ repodigest pack src/repodigest --entry cli.pack_command --budget 3000 --signatures-only

Example — XML output, written to a file for downstream tooling:

$ repodigest pack src/repodigest --entry graph.call_graph.CallGraph --format xml --output context.xml
$ head -6 context.xml
<context>
  <file path="src/repodigest/graph/call_graph.py">
    <symbol name="graph.call_graph.CallGraph" kind="class" tokens="631">
<![CDATA[
@dataclass
class CallGraph:

repodigest graph <path>

Print a Rich-rendered summary table of every module's classes/functions/methods, plus the total number of call-graph edges — a quick sanity check before packing, or a way to spot which modules are the most interconnected (and therefore most likely to pull in a lot of context via BFS).

$ repodigest graph src/repodigest
      RepoDigest graph summary: src/repodigest
┏━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━┓
┃ Module            ┃ Classes ┃ Functions ┃ Methods ┃
┡━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━┩
│ cli               │       0 │         3 │       0 │
│ graph.call_graph  │       1 │         2 │       6 │
│ packer.budget     │       4 │         1 │       4 │
│ packer.serializer │       0 │         4 │       0 │
│ packer.tokenizer  │       0 │         3 │       0 │
│ parsers.py_parser │       4 │        10 │       0 │
│ search.ranker     │       2 │         1 │       4 │
└───────────────────┴─────────┴───────────┴─────────┘
49 symbols across 7 modules, 56 call edges.

---

Programmatic Python API

Every layer is a plain, importable Python object — the CLI is a thin wrapper over the same calls you'd make from a script or a custom agent loop.

from pathlib import Path

from repodigest.graph.call_graph import CallGraph
from repodigest.packer.budget import ContextPacker, build_symbol_registry
from repodigest.packer.serializer import to_markdown, to_xml
from repodigest.search.ranker import SymbolRanker

root = Path("src/repodigest")

# Build once, reuse across multiple pack() calls — both are pure functions of the
# source tree, so there's no need to re-parse for every query in a session.
call_graph = CallGraph.from_directory(root)
registry = build_symbol_registry(root)

# Inspect the graph directly
print(call_graph.get_callers("packer.budget.ContextPacker.pack"))   # upstream
print(call_graph.get_callees("packer.budget.ContextPacker.pack"))   # downstream

# Pack a known symbol
packer = ContextPacker(root, budget=4000, signatures_only=True, call_graph=call_graph, registry=registry)
result = packer.pack("packer.budget.ContextPacker.pack")
print(f"{result.total_tokens}/{result.budget} tokens, {len(result.skipped)} symbols skipped")
print(to_markdown(result))

# Seed a pack target from a natural-language question — e.g. inside a custom
# agent loop that needs to find "the right code" before it can act on a task.
corpus = {qual: f"{qual} {record.signature_text}" for qual, record in registry.items()}
ranker = SymbolRanker(corpus)

target = ranker.top("Where is the token budget enforced?")
if target is not None:
    result = packer.pack(target)
    xml_context = to_xml(result)   # feed this directly into an LLM prompt

ContextPacker accepts an already-built call_graph/registry, so a long-running process (an agent server, a REPL) can amortize the one-time AST-parsing cost across many pack() calls against the same snapshot of the repository.

---

Test Verification & Benchmarks

./venv/bin/pytest tests/ -v

============================== 29 passed in 0.21s ==============================

Suite	Tests	Covers
tests/test_parser.py	6	Import/function/class/docstring extraction; full vs. signature-only rendering; decorator-line preservation in full mode
tests/test_call_graph.py	5	Cross-file edge construction; upstream/downstream lookups; unknown-symbol handling
tests/test_packer.py	6	BFS traversal order; strict budget enforcement (skip-not-truncate); file-entrypoint root resolution; unknown-target errors
tests/test_search.py	5	BM25 tokenization; relevance ranking; top_k; empty-corpus edge case
tests/test_cli.py	7	End-to-end pack/graph behavior across both output formats, --output, and error paths

All fixtures are synthetic, in-repo Python snippets (built via tmp_path) rather than mocks — every test exercises the real ast parser, the real call-graph builder, and the real BM25 math.

On "benchmarks": RepoDigest does not yet ship a dedicated performance benchmark suite (e.g. timing packing against a large real-world monorepo). The number above — the full 29-test suite running in ~0.2s on the author's machine — is the only currently-measured timing figure; treat it as a smoke-test data point, not a throughput guarantee. Formal benchmarking against larger codebases is open, tracked work.

---

Development & Solo Authorship

RepoDigest is authored and maintained by Nagendra Prasad Konakanchi (@nagendra-kon).

This is currently a solo project without a formal contribution process, but issues and pull requests are welcome:

• Bug reports / feature requests: open a GitHub issue with a minimal reproduction (a small Python snippet is usually enough, given how the test suite is structured).

• Pull requests: please keep changes scoped and include tests — every module in this repository (parsers, graph, packer, search, cli) has a corresponding tests/test_*.py file, and new behavior should extend the matching one rather than introduce a new pattern. Before opening a PR, run the full suite:

  ./venv/bin/pytest tests/ -v
  

• Coding conventions: the codebase favors small, single-purpose dataclasses and functions over inheritance hierarchies, and explicit from __future__ import annotations + X | None style type hints. Runtime dependencies are kept deliberately minimal: tiktoken (tokenization), click and rich (the CLI) are actively used by every module described above. requests is also declared in pyproject.toml from the project's initial scaffolding but is not yet imported anywhere — it's reserved for a future phase (e.g. calling a remote model API) rather than a currently-active dependency; if that phase doesn't materialize, it should be dropped.

Licensed under the MIT License.