candlekeep 2.1.0

RAG knowledge base server with ChromaDB and MCP interface

Candlekeep

The great library fortress on the Sword Coast, where all knowledge is preserved.

A RAG knowledge base server that gives AI agents the power to search, retrieve, and manage technical documentation through the Model Context Protocol. Ask a question, and the library answers — with the right scroll, expanded to full context, in milliseconds.

The Arcane Arts

• Bardic Knowledge — Documents are enriched with title and description at ingestion, woven into every embedding
• Bardic Inspiration — Result-time metadata boosting that ensures specific technical guides outrank generic content
• Arcane Recall — Intelligent expansion using Scholar's Discernment and Arcane Coalescence to return full sections without token waste
• Wild Magic — Hybrid retrieval merging Vector and BM25 (lexical) search, fixing "keyword blindness" for exact identifiers
• The Rosetta Seal — Corpus-derived BM25 token normalisation map that bridges surface-form variants (crossencoder ↔ cross-encoder), rebuilt automatically in the background after each ingest
• Divine Insight — Cross-encoder reranking for when precision matters more than speed
• The Relevance Ward — Results below a configured threshold are filtered, so the library says "I don't know" instead of guessing
• True Sight — Images in PDFs and markdown are captioned at ingestion via VLM, making diagram details searchable

Features

• Adaptive Search Routing: Three paths — hybrid (BM25+Vector), precise (Reranked), and explore (Divination — entity expansion)
• True Sight: Opt-in vision captioning for PDFs and markdown images — deployment topologies, benchmark charts, and architecture diagrams become searchable
• Statistical Rigor: Validated against The Centurion Set (100+ multi-category queries)
• Quality Gate: Documents must have frontmatter and structure to enter the library
• Embedding Protection: Auto-detects model mismatch on remote databases
• 14 MCP Tools: Search, ingest, critique, generate docs, agent memory, and more
• LLM & True Sight Providers: Pluggable anthropic, openai, bedrock, and openai_compat (Ollama/LM Studio/vLLM) — text and True Sight independently configurable
• Token Auth: Bearer token authentication for remote ChromaDB

Quick Start

PyPI (Recommended)

The easiest way to get the library up and running for use with any MCP client:

pip install candlekeep

# Run in stdio mode (standard)
candlekeep

# Run in HTTP mode (recommended for better performance)
CANDLEKEEP_TRANSPORT=http CANDLEKEEP_HTTP_PORT=8111 candlekeep

Docker (Isolated)

Run the server in a container. Note that if your ChromaDB is running on localhost, you'll need to use your host's internal IP (e.g., host.docker.internal on Docker Desktop):

docker run -p 8111:8111 \
  -e CHROMA_URL=http://host.docker.internal:8000 \
  ghcr.io/bansheeemperor/candlekeep:latest

Local Development

If you wish to contribute or modify the library's arcane secrets:

git clone https://github.com/raalgaw/candlekeep.git
cd candlekeep
pip install -e .
./scripts/setup.sh        # Download the tomes (embedding models)
./scripts/configure.sh    # Set your wards (configuration)
./scripts/start_chroma.sh  # Awaken the vault (ChromaDB)
candlekeep                # Enter the library

MCP Client Integration

HTTP mode (recommended) — one server, multiple agents. Models loaded once, shared memory, no cold-start per agent (~230ms first query vs ~6s in stdio mode):

# Start the server once
CANDLEKEEP_TRANSPORT=http CANDLEKEEP_HTTP_PORT=8111 candlekeep

{
  "mcpServers": {
    "candlekeep": {
      "url": "http://localhost:8111/mcp"
    }
  }
}

stdio mode — each agent spawns its own server process. Simpler setup, but each agent pays ~6s cold-start and loads its own copy of the models:

{
  "mcpServers": {
    "candlekeep": {
      "command": "/path/to/.venv/bin/candlekeep",
      "args": [],
      "env": {
        "CANDLEKEEP_SPICE": "true"
      }
    }
  }
}

See Setup Guide for auth configuration and production deployment.

The Tomes (Documentation)

• Setup Guide — Local and remote installation
• Authentication — Token configuration
• Architecture — System design and technical reference
• Design Decisions & Benchmarks — Why things are the way they are, with measured results
• Interactive Benchmark Chart — Visual comparison of paths
• Glossary of Retrieval — IR metrics explained in wizard sage style
• Research Diary — The full journey, every experiment, archived plans
• The Keeper's Chronicle — The story of how the library was built

MCP Tools

• search — Semantic search with adaptive routing (simple 22–36ms, precise ~1550ms)
• list_documents — List all indexed tomes
• get_stats — Library statistics
• critique_document — Check document quality before ingestion
• explore_entity — Explore an entity's co-occurring entities and source chunks via the graph
• generate_documentation — Scan a project and create structured docs
• memory_recall — Recall recorded memories semantically similar to a query
• memory_list — List recorded memories, newest first
• ingest — Add documents with automatic quality validation
• delete_document — Remove a tome from the index
• repopulate_database — Clear and rebuild the library
• rebuild_normalisation_map — Regenerate The Rosetta Seal from the current corpus after a full repopulate + ingest cycle
• memory_store — Record a short-form memory (lesson, failure pattern, debug tip) in the Chronicle
• memory_delete — Delete a memory from the Chronicle by ID

The Chronicle is a separate store of agent-recorded memories, isolated from the document corpus and preserved across repopulate_database.

Access to write tools is managed by your database permissions (configured via CHROMA_AUTH_TOKEN).

Testing

# Unit tests — no database required (~1.4s)
pytest tests/test_router.py tests/test_quality_gate.py tests/test_arcane_recall_unit.py \
       tests/test_protection.py tests/test_processor.py tests/test_search.py \
       tests/test_providers.py

# Benchmarks — requires local ChromaDB on localhost:8000
./scripts/start_chroma.sh
pytest tests/test_router_benchmark.py -v -s

59 unit tests covering router, quality gate, chunk expansion, embedding protection, document processing, and LLM/True Sight providers. Benchmark tests include regression assertions that fail if precision or content match drops below 80%.

Requirements

• Python 3.10+
• ChromaDB server (local or remote)

---

_{Candlekeep is a trademark of Wizards of the Coast. This project is unofficial fan content and is not endorsed by or affiliated with Wizards of the Coast.}