ContextCraft

Why ContextCraft?

Existing repository packers (repomix, gitingest, code2prompt) blindly concatenate every file verbatim. On real-world repositories, this blows past LLM context budgets, buries critical architecture inside thousands of lines of boilerplate (Lost-in-the-Middle degradation), and wastes API tokens.

ContextCraft is the AST-aware "MarkItDown for Codebases." Instead of naive concatenation or arbitrary character truncation, ContextCraft parses your codebase with Tree-sitter, ranks every file's architectural importance via Personalized PageRank on the cross-file dependency graph, and solves a Multiple-Choice Knapsack Problem (MCKP) across 4 AST fidelity tiers (FULL, SKEL, SIGS, TREE) to pack maximum architectural signal into your exact token budget $B$.

10-Second Quickstart

1. Installation (PyPI & uvx)

# Run instantly without installing via uvx:
uvx contextcraft-ai . --budget 16k --focus "AST elision and knapsack optimizer" -o context.md

# Or install globally / in your virtual environment from PyPI:
pip install contextcraft-ai
contextcraft . --budget 32k --format xml --copy

2. Command-Line Interface (CLI)

# Compress a repository into a strict 4,000-token budget biased toward your task
uv run contextcraft . --budget 4k --focus "AST elision and knapsack optimizer" -o context.md

# Emit Anthropic Claude-optimized <repository_context> XML and copy to clipboard
uv run contextcraft . --budget 32k --format xml --copy

3. Python SDK (Modeled after microsoft/markitdown)

from contextcraft import ContextCraft

cc = ContextCraft(budget_tokens="16k")
result = cc.compile("./my-repo", focus_query="authentication middleware")

print(result.markdown)
print(
    f"Compression: {result.stats.compression_ratio}x ({result.stats.compressed_tokens} / {result.stats.budget_tokens} tokens)"
)

4. Interactive Web Playground & "URL Swap Trick"

uv run uvicorn web.app:app --reload --port 8000

Open http://127.0.0.1:8000 to interactively tune token budgets, inspect PageRank scores, or use the URL Swap Trick: visit http://127.0.0.1:8000/<owner>/<repo>?budget=32k&focus=auth to automatically clone and compress any public GitHub repository!

Architecture Pipeline

┌────────────────────────────────────────────────────────────────────────────┐
│ 1. RepositoryScanner & SecretSanitizer                                     │
│    • Hierarchical .gitignore / .contextignore & binary filtering           │
│    • Pre-AST API key, PEM & high-entropy secret scrubbing (line-preserving)│
└────────────────────────────────────┬───────────────────────────────────────┘
                                     ▼
┌────────────────────────────────────────────────────────────────────────────┐
│ 2. UniversalASTParser (Tree-sitter + Deterministic Fallback)               │
│    • Extracts classes, functions, methods, signatures, and docstrings      │
│    • Computes exact post-docstring `body_range` spans for surgical elision │
└────────────────────────────────────┬───────────────────────────────────────┘
                                     ▼
┌────────────────────────────────────────────────────────────────────────────┐
│ 3. DependencyGraphScorer (Cross-File MultiDiGraph + Personalized PageRank) │
│    • Weighted edges from module imports & IDF-scaled symbol references     │
│    • BM25 query personalization (--focus) + 1-hop neighbor diffusion       │
└────────────────────────────────────┬───────────────────────────────────────┘
                                     ▼
┌────────────────────────────────────────────────────────────────────────────┐
│ 4. MultiResolutionElider (4 Discrete AST Fidelity Tiers per File)          │
│    • Tier 3 [FULL] (u=1.00): Complete sanitized source code                │
│    • Tier 2 [SKEL] (u=0.82): Signatures + types + docstrings (body elided) │
│    • Tier 1 [SIGS] (u=0.45): Imports & public declarations only            │
│    • Tier 0 [TREE] (u=0.00): Annotated directory tree entry only           │
└────────────────────────────────────┬───────────────────────────────────────┘
                                     ▼
┌────────────────────────────────────────────────────────────────────────────┐
│ 5. TokenBudgetOptimizer (Multiple-Choice Knapsack Solver) & Formatter      │
│    • Maximizes ∑ PageRank(i) · u(l_i) subject to TotalTokens ≤ Budget B    │
│    • Emits Markdown, Claude XML, or JSON with strict budget verification   │
└────────────────────────────────────────────────────────────────────────────┘

Quantitative Benchmark (ContextCraft Repository)

Evaluated using benchmarks/run_benchmark.py (--focus "AST elision knapsack optimizer pagerank"):

• Raw Uncompressed Repository (repomix / verbatim concatenation baseline): 43,151 tokens (1.00x)

Development & Verification

# Install all dependencies (core, web, and dev)
uv sync --locked --all-extras --dev

# Run lint, strict type checking, and test coverage
uv run ruff check src tests web benchmarks
uv run mypy src/contextcraft
uv run pytest --cov=contextcraft --cov-report=term-missing

License

Released under the MIT License.