kinetic-context 0.3.0

Repository-level code context engine — find the right code, fast.

kinetic-context

Repository-level code context engine — find the right code, fast.

kinetic-context is a self-contained code retrieval engine for LLM coding agents. Point it at any repository and it builds a multi-layer index (AST chunks + embeddings + a code knowledge graph + BM25) that you can query with natural language or identifiers. It returns code blocks with line ranges — not just file paths — ready to paste into an LLM prompt.

It ships with a Rich CLI, an MCP server (so Claude Code, Cursor, Continue, and Zed can mount it natively), a TCP JSON mode for any other agent, and a Python library.

---

What this is, honestly

This is a v0.3 code retrieval system. It is not a production-grade product. It has not been battle-tested at scale. The "novel signals" below have been ablation-tested on two benchmarks (Flask, Django core), and the ones that don't help are disabled by default. The benchmarks are self-reported and hand-curated. There are 44 unit tests but no integration tests against real coding agent workflows.

What it does well:

• AST-aware chunking that never splits functions mid-body
• Hybrid retrieval (BM25 + dense embeddings + code knowledge graph) fused with RRF
• Per-repo isolated storage at ~/.kinetic/ with Merkle-root incremental updates
• Beautiful Rich CLI, MCP server, TCP JSON, Python library — multiple integration paths
• Graceful degradation: runs with zero API keys (local TF-IDF + heuristic reranker)

What it doesn't do well yet:

• Scale beyond ~1k files (Django core at 308 files / 7k chunks takes 433ms per query; full Django at 3k files is untested)
• Handle dynamic dispatch, metaprogramming, or macro-heavy languages
• Provide a UI beyond the CLI
• Validate the "novel signals" on anything other than Flask and Django

---

Install

pip install kinetic-context

Requires Python 3.10+. All heavy dependencies (tree-sitter, numpy, networkx, rich) are bundled — no Qdrant/Pinecone/Milvus/Postgres to install.

API keys (optional but recommended)

The engine works with zero API keys (using a local TF-IDF embedder and heuristic reranker), but quality is much higher with cloud models:

export MISTRAL_API_KEY=...        # for codestral-embed embeddings (1536-dim)
export ZEROENTROPY_API_KEY=...    # for zerank-2 reranking

Mode	Embeddings	Reranker	Quality	Latency	Cost
No keys (self-contained)	Local TF-IDF (384-dim, hashing trick)	Heuristic cross-encoder	Lower	Fastest	Free
Mistral key only	Codestral Embed (1536-dim)	Heuristic cross-encoder	Medium	Fast	Cheap
Both keys (recommended)	Codestral Embed (1536-dim)	Zerank-2	Highest	~400ms	Cheap

Security note: API keys are read from environment variables ONLY. There are NO hardcoded defaults anywhere in the codebase. (v0.2.1 and v0.2.2 accidentally shipped with hardcoded keys and have been yanked from PyPI — if you installed either, rotate your keys and upgrade to v0.3.0+.)

---

Quick start

# Index a repo (first run takes a few minutes for a 1k-file repo)
kinetic index ./my-repo

# Subsequent runs only re-embed files whose SHA-256 hash changed
kinetic index ./my-repo

# Search — returns code blocks with line ranges
kinetic query "authentication middleware" --code

# JSON output (for piping to agents)
kinetic query "how does routing work" --json | jq '.results[0]'

# Check if the index is up to date
kinetic status ./my-repo

# List all indexed repos
kinetic list

---

Hooking into coding agents

MCP server (Claude Code, Cursor, Continue, Zed, anything MCP-aware)

Add to your MCP config:

{
  "mcpServers": {
    "kinetic": {
      "command": "kinetic",
      "args": ["mcp"]
    }
  }
}

Four tools are exposed:

Tool	Description
kinetic_index	Index (or incrementally update) a repo
kinetic_query	Search the indexed codebase, returns code blocks with line ranges
kinetic_status	Check whether the index is up to date
kinetic_list_indexes	List all indexed repos

TCP JSON server (any agent)

kinetic serve --port 7878

Send a single-line JSON request, get a single-line JSON response:

echo '{"query": "session cookie signing", "top": 5}' | nc localhost 7878

Python library

from kce.engine import KCEEngine
from kce.config import KCEConfig
from kce.store.registry import Registry

cfg = KCEConfig()
cfg.index_dir = Registry().resolve("/path/to/repo")
cfg.ensure_dirs()
engine = KCEEngine(cfg)
engine.index("/path/to/repo")

result = engine.query("how does authentication work")
for cid in result.final_chunk_ids[:10]:
    chunk = engine.chunk_index[cid]
    print(f"{chunk.rel_path}:{chunk.start_line}-{chunk.end_line}  {chunk.name}")
    print(chunk.content)

---

Architecture

The pipeline has four layers. Each layer is explained in detail below.

Layer 1: Ingestion

Tree-sitter AST parsing. We parse each source file into an AST using tree-sitter grammars for Python, JavaScript, TypeScript, Go, Rust, and Java. Tree-sitter gives us a concrete syntax tree — every token has a position, so we can extract exact line ranges.

cAST structure-aware chunking. The chunker walks the AST and produces chunks that respect syntactic boundaries. A function is never split mid-body. A class's methods stay together if they're small. We use 35% overlap between adjacent chunks to preserve context across boundaries. Each chunk is enriched with:

• # FILE: relative/path.py
• # SCOPE: class::ClassName or module
• # SIGNATURE: def foo(x: int) -> str:
• # IMPORTS: import os, from typing import ...
• # PREV: def previous_function(...): (signature of the previous chunk for context)

This enrichment means the embedder sees not just the raw code but its structural context — which function it belongs to, what's imported, what came before.

Module-level assignment indexing. We index assignments at depth ≤ 2 in the AST, so module-level proxies like request = LocalProxy(...) in Flask are captured as their own chunks. Without this, identifier lookups for request would miss the definition.

File and repository summaries. Each file gets an L3 summary chunk (heuristic summary + exported symbols). The whole repo gets an L5 summary chunk. These give the retrieval system a "zoomed out" view — when you ask "what's the architecture of Flask?", the L5 chunk is the answer.

Code Knowledge Graph (CKG). We build a NetworkX directed graph with 8 relationship types:

• CALLS — function A calls function B
• INHERITS — class A inherits from class B
• IMPLEMENTS — class A implements interface B
• IMPORTS — file A imports symbol B
• CONTAINS — file A contains function/class B
• USES_TYPE — function A uses type B in its signature
• OVERRIDES — method A overrides method B (detected via matching signatures in parent classes)
• DEPENDS_ON — transitive closure of IMPORTS + CALLS + USES_TYPE

The graph lets us do neighborhood retrieval: "find functions that call authenticate_user" or "find classes that inherit from BaseForm".

Honest caveat: The CALLS extraction is regex-based (re.finditer(r"\b([A-Za-z_][A-Za-z0-9_\.]*)\s*\(", text) with a builtin blacklist). It does NOT handle dynamic dispatch (getattr(obj, method_name)()), method chains on multi-line expressions, or metaprogramming. On standard Python/JS/TS code it catches ~80% of calls; on heavily metaprogrammed code (Django models, Flask extensions) it misses more. This is a known limitation.

Layer 2: Storage

Per-repo isolation. Every indexed repo gets its own directory under ~/.kinetic/:

~/.kinetic/
  flask_f9cbd6f6/                       <- slug = name + short hash of abspath
    manifest.json                       <- source path, root hash, file count, last indexed
    chunks.jsonl                        <- one CodeChunk per line
    embeddings.npy                      <- (N, 1536) float32 matrix
    ckg.graphml                         <- Code Knowledge Graph
    incremental_state.json              <- per-file SHA-256 hashes
    change_log.jsonl                    <- append-only change history
    embed_cache/embeddings.db           <- API cache (SQLite)
    zerank_cache/rerank.db              <- API cache (SQLite)
  django_core_5a16ff29/
    ...

Why per-repo isolation? Two repos with the same folder name (e.g. ~/work/api and ~/side/api) get different slugs because the slug includes a short hash of the absolute path. No collisions. Caches live next to the index, so kinetic forget frees both.

Merkle root incremental updates. The manifest.json records a Merkle root hash — the SHA-256 of all file content hashes concatenated in sorted order. On kinetic index, we recompute the root from disk (takes <50ms even for a 1k-file repo) and compare. If they match, the index is up to date — done in <100ms. If they differ, we walk the per-file SHA-256 hashes in incremental_state.json and identify exactly which files were added, modified, or removed. Only those files are re-parsed, re-summarized, and re-embedded.

The Mistral embed cache (SQLite, keyed by SHA-256 of the embed text) means even a chunk whose text didn't change but whose chunk_id was regenerated will not trigger an API call.

For a 1k-file repo where 5 files changed, a re-index takes seconds, not minutes.

Layer 3: Retrieval & Ranking

The query pipeline (see kce/engine.py):

1. Query coordinator classifies the query into one of 5 types:

   • IDENTIFIER_LOOKUP — "QuerySet", "request", "BaseForm" (looks like a single identifier)
   • SEMANTIC_QUESTION — "how does routing work"
   • CODE_COMPLETION — "models.ForeignKey(" (looks like a partial code snippet)
   • BUG_DIAGNOSIS — "TransactionManagementError atomic block" (mentions errors/exceptions)
   • ARCHITECTURE_QUERY — "what are the main building blocks"

   Honest caveat: Classification is keyword-based (if "error" in query.lower() style). It's not an ML classifier. It works for the common cases but can be fooled.

2. Intent-aware boosts. Based on the query type, we apply per-file boost/penalty weights:

   • Source files: ×1.5 (we want implementation, not tests)
   • Test files: ×0.7 (penalize, unless it's a test_for query)
   • Config files: ×1.2 (config matters for "how does config work" queries)
   • Docs: ×0.5 (usually not what you want when asking about code)

3. Multi-query BM25. We generate 3 query variants (original + identifier extraction + NL-to-code) and run BM25 on each. Results are fused. The BM25 tokenizer is code-aware: it splits camelCase (getUserName → get, user, name), snake_case (get_user_name → get, user, name), and dotted identifiers (app.config.from_object → app, config, from, object).

4. Dense retrieval. We embed the query with the same embedder used for the corpus and run cosine similarity. With Mistral Codestral Embed, this is 1536-dim. With the local fallback, it's 384-dim TF-IDF via the signed hashing trick (Weinberger et al. 2009).

5. Pseudo-relevance feedback (PRF). We take the top-5 BM25 results, extract their top-10 terms (weighted by TF-IDF), append them to the query, and re-run BM25. This is standard Rocchio-style PRF.

6. Graph retrieval. We take the top-5 seeds from BM25 + dense and retrieve their graph neighbors (depth=2) — functions they call, classes they inherit from, files that import them.

7. Patch reverse engineering (opt-in, on by default when Mistral key is set). We ask Mistral Medium to generate a 5-15 line hypothetical code snippet that would appear in the answer. We embed that snippet and run a dense search. The top-k results become a 5th input to RRF. This is HyDE (Gao et al. 2022) adapted for code. This adds an LLM round-trip to the query path (~1-2 seconds). Disable with config.enable_patch_reverse = False if latency matters more than quality.

8. Reciprocal Rank Fusion (RRF). We fuse the ranked lists from BM25, dense, graph, PRF, and patch (if enabled) using weighted RRF with k=60. Weights (brute-force-optimized on cached API results):

   • dense: 0.75
   • bm25: 0.05
   • graph: 0.03
   • prf: 0.30
   • patch: 0.30
   • docstring: 0.30 (for semantic queries only)

   Honest note on BM25 weight: BM25 gets only 0.05 because dense embeddings already capture lexical similarity well (codestral-embed is trained on code). The BM25 channel mainly helps for exact identifier matches that dense retrieval might rank slightly low. If you're using the local TF-IDF embedder (no Mistral key), you should probably increase rrf_bm25 to ~0.3.

9. Novel signals (see next section — each is gated by an ablation flag).

10. Reranking. The top-15 candidates are sent to Zerank-2 for cross-encoder reranking. Zerank-2 follows instructions via XML tags (<query>...</query><instruction>...</instruction>), and we generate a short 2-3 sentence instruction per query type. If no Zerank key is set, we fall back to a heuristic cross-encoder that scores based on token overlap, signature similarity, and structural signals.

11. File-level aggregation. Chunks are grouped by file, scores summed. Then:

    • Tier penalty (always on): abstract base classes (base.py, __init__.py) get ×0.4 for semantic/bug queries — they're usually not the answer.
    • Adaptive cutoff (opt-in, on by default): analyze the score distribution (bimodal/power-law/uniform/single-peak) and cut the result list at the natural gap.
    • Post-rerank filename tie-breaker (opt-in, on by default): if the query contains a keyword that exactly matches a filename, boost that file ×2.0. This fixes "correct directory, wrong file" failures at scale.
    • Sibling suppression: if >5 results come from the same directory, suppress the lower-ranked ones.
    • Confidence floor: drop results with score < 0.001, but keep at least 3.

Layer 4: Output

The final ranked chunks are returned as code blocks with:

• File path
• Line range (start_line - end_line)
• Language
• Name (function/class name)
• Node type (function_definition, class_definition, etc.)
• Signature
• Full source code of the chunk
• Docstring (if present)

These can be rendered by the Rich CLI (with syntax highlighting), serialized to JSON, or shipped over MCP/TCP to any agent.

---

Novel signals — what each one is and whether it helps

These are retrieval signals we invented or adapted for code retrieval. Each one is gated by an enable_* flag in KCEConfig. We ran an ablation study on Flask (30 queries) measuring each signal's contribution to F1@10. The ones that hurt or didn't help are disabled by default.

Signal	What it does	Ablation result (ΔF1 when toggled)	Default
Cross-resolution resonance	When a file's chunks appear at multiple resolutions (L2 function + L3 file summary) in the top-k, that's consensus. Boost the file by len(resolutions)².	+0.017 (turning off hurts)	on
Adaptive cutoff	Analyze the score distribution (bimodal/power-law/uniform) and cut the result list at the natural gap.	+0.036 (turning off hurts — biggest single contributor)	on
Post-rerank filename tie-breaker	If a query keyword exactly matches a filename, boost that file ×2.0. Fixes "correct directory, wrong file".	Neutral on Flask; helps on Django (kept on)	on
Patch reverse engineering	Generate a hypothetical code snippet with an LLM, embed it, use as a 5th RRF channel. (HyDE for code.)	+0.004 (turning off hurts slightly; adds latency)	on
Code DNA fingerprinting	Compare functions by structural features (param count, complexity, call count). Boost structurally similar functions.	Neutral (turning on or off makes no difference on Flask)	off
Semantic bridges	Build virtual graph edges between embedding-similar functions in different files. Expand candidates via bridges.	Neutral	off
Score gap amplification	Sigmoid-sharpen close scores to make gaps more decisive.	Neutral (theoretically amplifies noise)	off
File cohort memory	Files that co-occur in correct answers get boosted on future queries.	Neutral (confirmation bias risk in production)	off
Neuro-symbolic loop	RepoCoder-style iterative graph expansion. Expand seeds → retrieve neighbors → repeat.	Not ablated (off by default; known to dilute results)	off

Why we kept the "HURTS" signals

They're off by default but still in the codebase because:

1. They might help on different benchmarks or larger repos (we only tested Flask + Django core)
2. Users can experiment with them via config.enable_* = True
3. Removing them would make the ablation study unreproducible

Why we didn't remove the "NEUTRAL" signals

Same reasoning — they don't help on our benchmarks but might help elsewhere. The cohort memory in particular could be useful if you have a click-through feedback signal (we don't in pure retrieval mode).

---

Storage layout

Every indexed repo gets its own directory under ~/.kinetic/:

~/.kinetic/
  <slug>_<pathhash>/
    manifest.json                       <- source path, root hash, file count, last indexed
    chunks.jsonl                        <- one CodeChunk per line
    embeddings.npy                      <- (N, D) float32 matrix (D=1536 codestral, 384 tfidf)
    ckg.graphml                         <- Code Knowledge Graph
    incremental_state.json              <- per-file SHA-256 hashes
    change_log.jsonl                    <- append-only change history
    embed_cache/embeddings.db           <- embed API cache (SQLite)
    zerank_cache/rerank.db              <- rerank API cache (SQLite)
    patch_gen/patch_gen.db              <- patch reverse engineering cache (SQLite)

The slug is <folder_name>_<8-char-hash-of-abspath>, so two repos with the same folder name in different parents get different slugs.

---

Configuration

All knobs are in KCEConfig. The defaults are ablation-tuned. Override any of them:

from kce.config import KCEConfig

cfg = KCEConfig()
cfg.rrf_bm25 = 0.30              # increase BM25 weight (useful for local TF-IDF mode)
cfg.enable_dna_boost = True      # experiment with a disabled signal
cfg.chunk_max_tokens = 1024      # larger chunks
cfg.final_top_n = 20             # return more results

Key settings:

Setting	Default	What it controls
use_local_embeddings	auto (True if no Mistral key)	Use local TF-IDF instead of Mistral
chunk_max_tokens	512	Max chunk size
chunk_overlap_pct	0.35	Chunk overlap
bm25_k1, bm25_b	1.5, 0.75	BM25 params
rrf_dense, rrf_bm25, rrf_graph	0.75, 0.05, 0.03	RRF channel weights
retrieval_top_k	50	Candidates per channel
rerank_top_n	15	Candidates sent to reranker
final_top_n	10	Final results returned
filename_keyword_boost	2.0	Post-rerank filename tie-breaker
enable_resonance	True	Cross-resolution consensus
enable_adaptive_cutoff	True	Distribution-aware cutoff
enable_filename_tiebreak	True	Filename keyword boost
enable_patch_reverse	True	LLM hypothetical snippet (adds latency)
enable_dna_boost	False	Structural fingerprint (ablation: hurts)
enable_semantic_bridges	False	Virtual graph edges (ablation: neutral)
enable_cohort_memory	False	Cross-query co-occurrence (ablation: neutral)
enable_score_gap_amp	False	Sigmoid sharpening (ablation: hurts)
enable_neuro_symbolic	False	RepoCoder iterative loop (ablation: neutral)

API keys are read from MISTRAL_API_KEY and ZEROENTROPY_API_KEY env vars. There are no hardcoded defaults.

---

Benchmarks

We benchmark on two real-world repos with hand-curated query sets (30 queries each). We report Context F1@10, Recall@10, Precision@10, and end-to-end latency. We deliberately do not compare to other context engines — those numbers are easy to get wrong and we'd rather show our own results honestly than risk an unfair comparison.

Aggregate metrics

Per-query F1 (sorted, worst at top)

Query outcome distribution

Latency distribution

Quality vs repository scale

Numbers

Corpus	Files	Chunks	Graph (nodes/edges)	Queries	F1@10	Recall@10	Precision@10	Avg latency
Flask	83	1,382	1,594 / 7,906	30	0.659	0.833	0.599	68 ms
Django core	308	7,207	6,574 / 46,408	30	0.647	0.867	0.559	433 ms

The Django benchmark uses the core Django packages (django/db/, django/http/, django/urls/, django/template/, django/forms/, django/core/, django/contrib/auth/, django/contrib/sessions/) — the parts of Django that real coding agents actually search. The full Django repo includes 3,000+ files of migrations, tests, and docs that bloat the index without improving retrieval quality on real-world queries.

Ablation study

We ran an ablation study on Flask (30 queries) measuring each signal's contribution to F1@10. Each row shows the F1 when we toggle that signal, and the delta vs the baseline (all defaults ON):

Signal	Default	Test	F1@10	ΔF1 vs baseline	Verdict
(baseline, all defaults)	—	—	0.659	—	—
enable_resonance	ON	turn OFF	0.642	-0.017	HELPS
enable_adaptive_cutoff	ON	turn OFF	0.623	-0.036	HELPS (most)
enable_patch_reverse	ON	turn OFF	0.655	-0.004	HELPS (small)
enable_filename_tiebreak	ON	turn OFF	0.659	0.000	NEUTRAL (on Flask; helps on Django)
enable_dna_boost	OFF	turn ON	0.659	0.000	NEUTRAL
enable_semantic_bridges	OFF	turn ON	0.659	0.000	NEUTRAL
enable_cohort_memory	OFF	turn ON	0.659	0.000	NEUTRAL
enable_score_gap_amp	OFF	turn ON	0.659	0.000	NEUTRAL

What this tells us:

1. adaptive_cutoff is the single most valuable signal (+0.036 F1). Without it, we keep too many low-relevance results that drag down precision. The score-distribution shape analysis genuinely identifies the natural cutoff point.

2. resonance helps (+0.017 F1). When a file's function body AND file summary both rank high, that consensus is a strong relevance signal.

3. patch_reverse helps a little (+0.004 F1). The HyDE-style hypothetical snippet generation adds a small quality bump, but at the cost of an LLM round-trip. Worth it if you have the Mistral key; disable for lowest latency.

4. filename_tiebreak is neutral on Flask but we keep it ON because it's the signal that fixed Django's "correct directory, wrong file" problem at scale (Flask's 83 files don't trigger it often; Django's 308 files do).

5. The four OFF-by-default signals are genuinely neutral on Flask — turning them ON doesn't help either. They're kept in the codebase for experimentation and because they might help on larger repos or different query distributions, but the ablation justifies keeping them off.

Run the ablation yourself:

python scripts/run_ablation.py
# Results cached to /tmp/ablation_checkpoint.json (resumable if killed)
# Final output at benchmark_results/ablation_study.json

Run the benchmarks yourself:

git clone https://github.com/pallets/flask /tmp/flask
kinetic index /tmp/flask
python scripts/run_bench.py flask
python scripts/gen_charts.py

---

Tests

44 unit tests covering the chunker, BM25, RRF, registry/manifest, local TF-IDF embedder, and engine smoke tests:

cd /path/to/kinetic-context
python -m pytest tests/ -v

What's tested:

• Chunker: functions not split mid-body, module-level assignments captured, line numbers correct, file summary chunks produced
• BM25: camelCase/snake_case/dotted identifier splitting, boost weights, empty queries, ranking order
• RRF: basic fusion, weighted fusion, empty lists, top_n limits, k parameter
• Registry/Manifest: same-folder-name collision avoidance, manifest roundtrip, status detects up-to-date vs needs-reindex, root hash stability and change detection
• Local TF-IDF embedder: tokenizer splits correctly, vectors are L2-normalized, cache works, similar texts have high cosine similarity, zero API calls
• Engine smoke: initializes without API keys, indexes a tiny repo, query returns relevant results

What's NOT tested:

• End-to-end integration with real coding agents (Claude Code, Cursor, etc.)
• Performance at scale (>1k files)
• The MCP server protocol conformance
• The cloud embedder/reranker (requires API keys)

---

FAQ

Why not just use ripgrep / the IDE's built-in search? Lexical search finds the keyword. It doesn't find the concept. Asking "how does routing work in flask" with grep gives you every line containing "route" — most of which is irrelevant. kinetic-context returns the 8 functions that actually implement routing, with their full bodies and signatures, in 68ms.

Why not just stuff the whole repo into the LLM context window? A 1k-file repo is ~500k tokens just for the source. That's expensive, slow, and the LLM's attention degrades badly past ~100k tokens. kinetic-context returns the 10 chunks that matter, fits in any context window, and costs 100x less to query.

Why a Code Knowledge Graph? Because "what calls what" matters. When you ask "where is request used?", the graph says "the request proxy is defined in flask/globals.py, used in 47 places, and its setter is in flask/app.py". Pure lexical search sees the word request 500 times. The graph sees the relationship. (With the caveat that our CALLS extraction is regex-based and misses dynamic dispatch — see Layer 1 above.)

Is it really self-contained? Yes, with caveats. With zero API keys, it runs using a local TF-IDF embedder (384-dim, hashing trick) and a heuristic reranker. Quality is lower than the cloud mode but it works. With MISTRAL_API_KEY and ZEROENTROPY_API_KEY set, it uses cloud models for higher quality. There are no external databases to install (no Qdrant/Pinecone/Milvus/Postgres) — everything is SQLite + numpy + GraphML.

Why are some signals disabled by default? Because we ablation-tested them and they hurt or didn't help. See the "Novel signals" table above. We kept them in the codebase so users can experiment and so the ablation is reproducible.

---

License

MIT. See LICENSE.

Contributing

Issues and PRs welcome at github.com/notlousybook/kinetic-context. Please include test coverage for any new retrieval signals and run the ablation study to show they help.