LLM Wiki Monorepo

A knowledge base that builds itself.
AI agents read your documents, build a structured wiki, and keep it current — no database, no API lock-in, one repo.

What is this? • Features • Quick Start • Architecture • Packages • Templates • Credits • License

pip install baissarienterprises-llm-wiki

What is this?

LLM Wiki is a production-grade knowledge engine that turns raw documents into a living, cross-linked Markdown wiki. Instead of re-retrieving documents on every query (RAG), the system incrementally builds and maintains a persistent knowledge base. Sources are compiled once, kept current, and compound over time.

It's a monorepo of everything needed to run a self-building wiki: a Python CLI package, an MCP server for programmatic access, a knowledge graph engine with community detection, a Chrome web clipper, a web viewer, an Obsidian plugin, and 20 domain templates — all wired together through an agent skill that works with any LLM.

Features

• Two-Step Ingest — LLM analyzes sources first, then generates structured wiki pages. SHA256 caching skips unchanged files. Streaming progress, multi-provider support (OpenAI, Anthropic, DeepSeek), and agent-native mode that needs zero API keys.

• Agent-Native Provider — route ingest through Hermes, Claude Code, or Codex directly. No external API keys required when running inside an AI agent session.

• Structured Output — Pydantic-typed parsing via instructor. No regex guesswork. Retry with exponential backoff on transient failures. Token counting and cost estimation per operation.

• Concurrency Control — per-page advisory locking, atomic writes (temp → fsync → rename), SHA256 conflict detection, and three-tier conflict management with automatic cleanup. Multiple agents can safely operate on the same wiki.

• Knowledge Graph Engine — Louvain community detection with full Blondel et al. modularity. 4-signal relevance model with precomputed adjacency. Surprising connection discovery and knowledge gap detection. Pure Python fallback included.

• 16-Pass Automated Lint — dead links, orphans, frontmatter validation, contradictions, source drift, unresolved conflicts, and stale page detection when raw sources change.

• SQLite FTS5 Search — full-text search with SHA256 freshness detection. Pre-builds at startup, incremental updates, BM25 fallback. Rebuild with --rebuild.

• Inverted Entity Index — dual-map entity→pages + page→entities for O(1) link suggestions. 4-signal scoring with automatic wikilink insertion.

• MCP Server — 14 stdio tools for programmatic wiki access. Direct Python sidecar with zero subprocess overhead. Integrates with Claude Desktop, Codex, Cursor, and any MCP-compatible client.

• Community Verification Suite — NMI/ARI cross-validation across 5 seeds, statistical similarity metrics, modularity tolerance within 1% relative error.

• Backup & Recovery — tar.gz snapshots with restore, integrity verification, and automatic pruning. One-command --auto for safe state.

• 20 Domain Templates — research, codebase, finance, machine learning, cybersecurity, medicine, algorithmic trading, and more. Every template ships with PURPOSE.md, SCHEMA.md → CLAUDE.md, and extra-dirs.json.

• Deep Research — web search → fetch → ingest → synthesize. Multi-source compilation into structured wiki pages.

• Chrome Web Clipper — one-click web page capture with Readability + Turndown, auto-triggering ingest after clip.

• CI/CD Pipeline — pytest + vitest matrix across Python 3.10–3.12 and Node 18–22, coverage reporting, caching, benchmark artifacts.

Quick Start

# Install from PyPI
pip install baissarienterprises-llm-wiki

# Or install from source
git clone https://github.com/JeanBaissari/llm-wiki-monorepo.git
cd llm-wiki-monorepo
bash install.sh

# Scaffold a wiki
llm-wiki scaffold ~/my-wiki "My Research" --template research

# Ingest a source (two-step agent loop)
llm-wiki ingest ~/my-wiki raw/articles/my-source.md

# Use agent-native provider (no API keys — inside Hermes/Claude Code/Codex)
llm-wiki ingest ~/my-wiki raw/articles/my-source.md --llm opencode

# Check quality
llm-wiki lint ~/my-wiki

# Clean up old conflicts automatically
llm-wiki lint ~/my-wiki --clean-conflicts

# Build search index
llm-wiki index ~/my-wiki

# Discover hidden connections
llm-wiki insights ~/my-wiki

# Health check
llm-wiki health ~/my-wiki

# Start MCP server
llm-wiki serve ~/my-wiki

Architecture

wiki/ directory  ← shared state (Markdown files)
     │
     ├── Agent Skill + Python Scripts   → 20+ scripts: scaffold, ingest, lint,
     │                                     discover, insights, backup, link-suggest,
     │                                     deep-research, audit, benchmark, sidecar,
     │                                     lock_wiki, atomic_write, content_hash,
     │                                     index_wiki, louvain, health_check, wiki_logging
     ├── LLM Provider Layer             → openai, anthropic, litellm, opencode
     │   (src/llm_wiki/ + skill/scripts/providers/)
     ├── MCP Server (stdio)             → programmatic access, 14 tools,
     │                                     direct sidecar (zero subprocess)
     ├── Graph Engine (Node.js)         → relevance model, Louvain, insights,
     │                                     graphology bridge, verification suite
     ├── Web Viewer + Obsidian Plugin   → human browsing + feedback
     ├── Browser Extension              → web clipping + auto-ingest
     └── templates/                     → 19 domain schemas

Packages

Templates (20 domains)

research codebase finance algorithmic-trading algorithmic-trading-mql4 cybersecurity machine-learning prompt-engineering copywriting marketing design-systems architecture crypto commodities decompilers medicine developer-tools personal-growth reading business

Every template provides: PURPOSE.md (scope + goals), SCHEMA.md → CLAUDE.md (page types, conventions, frontmatter, cross-referencing, contradiction handling), extra-dirs.json (domain directories).

Documentation

Requirements

• Python 3.10+ — for all skill scripts and PyPI package
• Node.js 18+ — for MCP server, graph engine, web viewer
• npm — for TypeScript package management
• pip dependencies — openai, anthropic, litellm, instructor, tenacity, tiktoken, python-dotenv, pydantic, portalocker (auto-installed via pip install)

Credits

The foundational methodology comes from Andrej Karpathy's llm-wiki pattern — using LLMs to incrementally build and maintain a personal wiki from raw sources. This project is a production-grade implementation with concurrency control, multi-provider LLM support, FTS5 search, MCP integration, and community detection.

Also see the desktop application fork by nashsu — a cross-platform Tauri app with graph visualization, vector search, and a rich chat interface built on the same foundational pattern.

License

MIT