ctxforge — compile raw knowledge into agent-ready context

The context-layer fix. Instead of wiring raw documents straight into an agent's context window (slow, noisy, pattern-matched guesswork), ctxforge compiles a source directory into a structured wiki: per-document summaries, keyword lists, TF-IDF concept clusters with backlinks, a health-check linter, and a small search engine an agent can query over a CLI.

Why

• Context is the bottleneck, not the model. Karpathy: "a large fraction of my recent token throughput goes into manipulating knowledge" — raw sources compiled into a wiki with summaries, backlinks, and concept articles, queried through a CLI.
• Vector retrieval breaks on vocabulary mismatch ("why did the deploy fail" vs get_pipeline_run_logs). TF-IDF keyword retrieval over compiled summaries keeps agents on-target without an embedding service or vector DB.
• Health checks catch decay: stale sources, missing summaries, orphan docs — the wiki is treated as a product with an update cadence, not a setup step.

Usage

./ctxforge compile ~/notes -o ~/wiki          # build wiki (extractive, $0, offline)
./ctxforge compile ~/notes -o ~/wiki --llm    # LLM-enriched summaries (optional)
./ctxforge search ~/wiki "reservoir pressure decline"   # ranked results
./ctxforge health ~/wiki                       # stale / orphan / missing-summary lint
CTXFORGE_AGENT_JSON=1 ./ctxforge search ~/wiki "deploy failed"  # JSON for agents

LLM enrichment is optional and model-agnostic (any OpenAI-compatible endpoint):

export OLLAMA_URL=http://localhost:11434/v1/chat/completions   # local
export CTXFORGE_LLM_URL=https://api.openai.com/v1/chat/completions
export CTXFORGE_LLM_KEY=sk-...
export CTXFORGE_LLM_MODEL=llama3.1

Output

wiki/
  index.json        # machine-readable: docs, summaries, keywords, concepts, idf
  docs/<slug>.md    # one per source: summary + keywords + backlinks
  concepts/concept-N.md  # TF-IDF clusters with defining terms + source links

Search returns ranked paths + summaries; with CTXFORGE_AGENT_JSON=1 it emits a single JSON line an agent can consume directly.