tokenkeeper 0.1.0

Local RAG memory for Claude Code -- reduce prompt tokens by 80%

TokenKeeper

Local RAG memory for Claude Code. Reduce prompt token consumption by ~80% on knowledge-heavy projects.

TokenKeeper is an MCP server that indexes your project's documents and code, then exposes semantic search tools to Claude Code. Instead of loading entire files into context, your agents query for only the relevant chunks.

The Problem

On a project with 34 phases of planning docs, a single agent cycle loads 141K tokens (70% of context) just for background knowledge — before it starts working. Quality degrades as context fills up.

The Solution

TokenKeeper replaces "load everything" with "query what's relevant":

	Traditional	With TokenKeeper
Prompt tokens	141,345	26,959
Context used	70.7%	13.5%
Tokens saved	—	114,386 (80.9%)

Your agents stay in the high-quality zone of their context window.

How It Works

Your project files
    |
    v
[Indexer] --> Chunks with embeddings --> ChromaDB (persistent vectors)
                                              |
                                              v
Claude Code agent --> search_knowledge("topic") --> Top-k relevant chunks

• Hybrid search — semantic similarity (vector) + keyword matching (BM25), merged via Reciprocal Rank Fusion
• Local-first — Ollama for embeddings, ChromaDB for storage. No cloud, no API keys required
• Auto-indexing — file watcher detects changes and re-indexes automatically
• Per-project isolation — each project gets its own .rag/ directory

Quick Start

Package name: TokenKeeper is the project brand name. The PyPI package is tokenkeeper:

pip install tokenkeeper

Until published to PyPI, install from source with uv sync.

Prerequisites

• Python 3.10+
• Ollama installed and running
• uv (Python package manager)

Install

git clone https://github.com/admin-sosys/TokenKeeper.git
cd TokenKeeper
uv sync
ollama pull nomic-embed-text

Add to Any Project

Create .mcp.json in your project root:

{
  "mcpServers": {
    "tokenkeeper": {
      "command": "/path/to/TokenKeeper/.venv/bin/python",
      "args": ["-m", "tokenkeeper"],
      "env": {
        "TOKENKEEPER_PROJECT": "${workspaceFolder}"
      }
    }
  }
}

Windows: Use .venv\Scripts\python.exe instead of .venv/bin/python

Start (or restart) Claude Code in that project. TokenKeeper will:

1. Create a .rag/ directory for index data
2. Index all markdown, JSON, and code files
3. Expose 4 MCP tools for search and management

Add .rag/ to your project's .gitignore.

Verify

Ask Claude Code:

Check the indexing status

Then test a search:

Search the knowledge base for "authentication flow and session management"

MCP Tools

Tool	Purpose
search_knowledge	Hybrid semantic + keyword search across indexed content
indexing_status	Check if indexing is complete, in progress, or failed
reindex_documents	Trigger manual reindexing (all or specific files)
get_index_stats	Index statistics — file count, chunk count, timestamps

search_knowledge Parameters

Param	Type	Default	Description
query	string	required	Natural language search query
top_k	int	10	Results to return (1-50)
alpha	float	0.5	Hybrid weight: 0.0 = keyword only, 1.0 = semantic only
mode	string	"hybrid"	"hybrid", "semantic", or "keyword"

Configuration

TokenKeeper auto-creates .rag/.rag-config.json on first run:

{
  "content_mode": "docs",
  "chunk_size": 1000,
  "overlap": 200,
  "alpha": 0.5,
  "mode": "hybrid",
  "watch_enabled": true,
  "debounce_seconds": 3.0
}

Setting	Default	Description
content_mode	"docs"	"docs" (md/json), "code" (source files), or "both"
chunk_size	1000	Characters per chunk (100-10000)
overlap	200	Character overlap between chunks
alpha	0.5	Hybrid search weight
mode	"hybrid"	Search strategy
watch_enabled	true	Auto-reindex on file changes

Architecture

TokenKeeper/
  src/tokenkeeper/
    server.py          # FastMCP server + lifespan
    indexer.py         # Discovery -> ingestion -> embedding -> storage
    search.py          # Hybrid search with RRF fusion
    embeddings.py      # Ollama (local) or Google Gemini (cloud)
    storage.py         # ChromaDB persistent client
    bm25_index.py      # BM25 keyword index
    watcher.py         # File system monitoring with debounce
    config.py          # Pydantic configuration
    health.py          # Startup health checks

Stack: Python 3.10+ | FastMCP | ChromaDB 1.5.0 | Ollama | BM25

Embedding Providers

Ollama (Default, Local)

• Model: nomic-embed-text (768 dimensions)
• No API key needed
• Runs on CPU (no GPU required)

Google Gemini (Optional, Cloud)

• Model: gemini-embedding-001 (3072 dimensions)
• Requires GOOGLE_API_KEY environment variable
• Higher quality embeddings, but requires internet

File Types Indexed

Mode	Extensions
"docs"	.md, .mdx, .json
"code"	.ts, .tsx, .js, .jsx, .py, .mjs, .go, .rs, .java, .rb, .c, .cpp, .h
"both"	All of the above

Always excluded: node_modules/, .git/, .next/, __pycache__/, .rag/, dist/, build/

Performance

Metric	Value
First index (500 files)	~3-5 minutes
Subsequent startups	~5 seconds (cached)
Search latency	~150ms per query
Storage	~100-200 MB per 2000-file project

Testing

# All tests (skip Ollama-dependent if not running)
uv run pytest tests/ -v --tb=short

# Token savings benchmark
uv run pytest tests/test_practical_token_savings.py -v -s

# Agent comparison (RAG vs traditional)
uv run pytest tests/test_agent_comparison.py -v -s

Troubleshooting

Issue	Fix
"Ollama connection refused"	Run ollama serve to start the server
"nomic-embed-text not found"	Run ollama pull nomic-embed-text
Claude Code doesn't show RAG tools	Ensure .mcp.json is in project root, restart Claude Code
0 chunks indexed	Check TOKENKEEPER_PROJECT env var points to your project root
Slow first index	Normal — subsequent starts load cached ChromaDB in ~5 seconds
Search returns irrelevant results	Try mode: "keyword" or lower alpha to 0.3

Docs

• QUICKSTART.md — Setup, toggling, A/B testing, GSD workflow integration
• IMPLEMENTATION-GUIDE.md — Architecture deep dive, cost analysis, integration patterns

License

MIT