llmwikify

Build persistent, LLM-maintained knowledge bases — and reproduce quant research from papers.

llmwikify is a Python CLI + library + unified server for building persistent, LLM-maintained knowledge bases with a dedicated quant research reproduction pipeline.

⚠️ Beta Release — APIs may shift between minor versions. Report issues on GitHub.

---

Why llmwikify?

🔍 Smart Search	SQLite FTS5 + optional QMD hybrid (BM25 + vector + LLM reranking)
🔗 Bidirectional Links	Automatic [[wikilink]] detection with section-level granularity
🧠 Knowledge Graph	8 relation types, PageRank, community detection, interactive D3.js visualization
🤖 ReAct Agent	Streaming chat with tool calling, confirmations, and 26 MCP tools
📊 Quant Pipeline	Paper → 6-layer Factor YAML → DuckDB → Backtest → L5 reflection
🌐 Unified Server	MCP + REST + WebSocket + Web UI in one process

---

Features

Chat + ReAct Agent ⚠️ Under Active Development

Streaming chat with tool calling, confirmations, and 26 MCP tools. The agent can search your wiki, analyze sources, and generate insights — all with human-in-the-loop confirmations.

⚠️ Note: This feature is under active development and may be unstable.

Markdown Editor

Split-pane live markdown editor with page tree, front-matter panel, and wikilink autocomplete. Edit, preview, and manage your wiki pages in one view.

Knowledge Graph

Interactive D3.js force-directed graph with PageRank node sizing, community coloring, and bridge highlighting. Explore relationships between your wiki pages visually.

Dashboard

Track your knowledge growth with metrics cards, Wiki Dream activity timeline, and health indicators. See how your wiki evolves over time.

---

Quick Start

# Install (with PDF/document extractors + web server)
pip install 'llmwikify[extractors,web]'

# Create and enter wiki directory
mkdir my-wiki
cd my-wiki

# Initialize a wiki
llmwikify init

# Add a page manually
llmwikify write_page "hello" --content "# Hello World\n\nThis is my first page."

# Ingest a markdown source (PDF requires [extractors])
llmwikify ingest README.md

# Build the link index (finds [[wikilinks]])
llmwikify build-index

# Search the wiki
llmwikify search "hello"

# Read a page
llmwikify read_page hello

# Show wiki status
llmwikify status

# Start the server (MCP + REST + Web UI)
llmwikify serve --web --port 8765
# Open http://localhost:8765

LLM Setup (for AI features)

# Option 1: One-shot — set up LLM with your wiki
export OPENAI_API_KEY=sk-...
llmwikify init --llm

# Option 2: Standalone — set up LLM separately
llmwikify init-llm
llmwikify init-llm --provider openai --api-key sk-...
llmwikify init-llm --provider anthropic  # uses ANTHROPIC_API_KEY
llmwikify init-llm --provider minimax    # uses MINIMAX_API_KEY

# Option 3: Interactive — init will prompt you
llmwikify init
# 💡 LLM features (analyze-source, synthesize, chat) need ~/.llmwikify/llmwikify.json
#    No LLM config detected. Set one up now? [y/N]: y
#    Detected OPENAI_API_KEY in env vars. Provider: openai
# ✅ LLM config written to ~/.llmwikify/llmwikify.json

Custom endpoint / OpenAI-compatible

llmwikify init-llm \
    --provider openai \
    --api-key sk-... \
    --base-url https://api.deepseek.com/v1

Non-interactive (CI / scripts)

llmwikify init --llm --no-llm-prompt --llm-overwrite

Without LLM: init, search, write_page, read_page, build-index, references, lint, graph-analyze, export-graph all work offline.

---

Doctor — Health Check

llmwikify doctor                     # Check everything (5s LLM test)
llmwikify doctor --skip-llm          # Skip LLM API call (faster)
llmwikify doctor --wiki-root /path   # Check a specific wiki
llmwikify doctor --json              # JSON output for CI/scripts

What it checks

#	Check	What
1	Config	~/.llmwikify/llmwikify.json exists, parseable, has valid api_key
2	Python	Version >= 3.10
3	Core deps	llmwikify, yaml, duckdb, jinja2
4	Optional extras	fastapi, fastmcp, watchdog, networkx, markitdown, tiktoken, httpx
5	LLM connectivity	Actually calls provider API (5s timeout) with "Say hi" — verifies key works
6	Wiki directory	wiki.md, .llmwikify.db, index.md, raw/ present
7	Permissions	~/.llmwikify/ and wiki root are writable
8	WebUI bundle	ui/webui/dist/index.html exists
9	Server	GET /api/health returns 200

Exit codes

Code	Meaning
0	All checks passed
1	One or more checks failed
2	Config missing — run llmwikify init-llm

Examples

# CI integration (JSON + skip LLM)
llmwikify doctor --json --skip-llm | jq -e '.summary.failed == 0'

# Quick check before running expensive operations
llmwikify doctor --skip-llm  # ~1 second

---

Tutorial

New to llmwikify? Start with our 5 end-to-end scenarios:

#	Scenario	Description
1	Personal Reading Notes	PDF → searchable wiki with cross-references
2	Company Due-Diligence KB	Multi-source analysis → knowledge graph
3	Multi-Wiki Collaboration	Manage multiple wikis through one server
4	Chat + ReAct Agent	LLM-powered Q&A with tool calling
5	Quant Reproduction	Paper → Factor → Backtest → L5 reflection

📖 Full tutorial: docs/TUTORIAL.md (40-60 min read) 🎯 Runnable examples: examples/ (8 playbooks, no LLM required)

---

Features at a Glance

Feature	Description
Wiki Core	FTS5 search, bidirectional references, query compounding, multi-wiki registry
Smart Lint	Broken links, orphans, contradictions, outdated pages, knowledge gaps
Knowledge Graph	8 relation types, PageRank, community detection, HTML/SVG/GraphML export
Chat + Agent	ReAct streaming, 26 MCP tools, skills system, research engine
Quant Reproduction	Paper extraction, 6-layer factors, DuckDB, backtesting, L5 reflection
Web UI	React SPA: editor, graph, dashboard, chat, quant pages
Extraction	PDF, Word, Excel, PowerPoint, images, audio, web, YouTube
MCP Server	26 tools over stdio + HTTP, multi-wiki support

---

Architecture

graph TB
    subgraph "Interfaces"
        CLI[CLI]
        MCP[MCP Server]
        REST[REST API]
        UI[Web UI]
    end
    
    subgraph "Apps"
        Wiki[Wiki Service]
        Chat[Chat + ReAct]
        Research[Research Engine]
        Agent[Agent Runtime]
    end
    
    subgraph "Kernel"
        Engine[Wiki Engine]
        Graph[Knowledge Graph]
        Search[Search Engine]
        Storage[SQLite Storage]
    end
    
    subgraph "Foundation"
        LLM[LLM Client]
        Extract[Extractors]
        Config[Configuration]
    end
    
    subgraph "Reproduction"
        Paper[Paper Pipeline]
        Factor[Factor Library]
        Backtest[Backtest Engine]
    end
    
    CLI --> Wiki
    MCP --> Wiki
    REST --> Wiki
    UI --> Chat
    Chat --> Engine
    Research --> LLM
    Paper --> LLM
    Factor --> Storage
    Backtest --> Factor

---

Installation

pip install llmwikify              # Core (zero hard deps)
pip install llmwikify[all]         # Full features
pip install llmwikify[web]         # Web UI + REST
pip install llmwikify[mcp]         # MCP server
pip install llmwikify[extractors]  # PDF/Office/media

Optional Extras

Extra	Purpose
extractors	PDF / Office / images / audio / YouTube via MarkItDown
mcp	MCP server (fastmcp)
watch	Filesystem watching (watchdog)
graph	Graph visualization + community detection
web	FastAPI / Starlette / Uvicorn for the unified server
agent	Scheduler + filelock + DuckDuckGo / Tavily search
llm	tiktoken for token counting
all	Everything above

---

CLI Reference

Core Wiki Operations

Command	Description
init	Initialize a wiki (creates dirs + wiki.md schema + .llmwikify.db)
init --llm	Initialize wiki + set up LLM config (one-shot)
init-llm	Set up LLM config only (auto-detects from OPENAI_API_KEY etc.)
write_page	Write/update a wiki page (llmwikify write_page "Name" --content "...")
read_page	Read a wiki page (name only, no .md suffix)
ingest	Ingest a source file or URL (PDF requires [extractors])
batch	Batch ingest a directory of sources
status	Show wiki stats (page count, index, links)
log	Record an operation in log.md

Search & Analysis

Command	Description
search	Full-text search (FTS5 backend)
analyze-source	LLM-powered source analysis and caching
knowledge-gaps	Detect missing pages, outdated content, redundancy
suggest-synthesis	LLM: generate cross-source synthesis suggestions
synthesize	Save query answer as a wiki page
report	Generate unexpected connections report

Link Index & References

Command	Description
build-index	Build bidirectional link index from [[wikilinks]]
references	Show inbound/outbound links, broken links, stats
fix-wikilinks	Fix broken wikilinks by adding directory prefix

Knowledge Graph

Command	Description
graph-analyze	PageRank, community detection, suggestions
graph-query	Query graph (neighbors / path / stats / context)
community-detect	Detect knowledge communities (Leiden/Louvain)
export-graph	Export visualization (HTML/SVG/GraphML)

Multi-Wiki

Command	Description
wikis	Multi-wiki management (list / add / remove / scan)

Server & MCP

Command	Description
serve	Start unified server — MCP + REST + Web UI (alias: mcp)
watch	Watch raw/ directory for new files, auto-ingest

Quant Research

Command	Description
quant-init	Scaffold quant/ directory structure
reproduce	Paper reproduction pipeline (Stage 0/1 + Track A/B)

Database & Health

Command	Description
db	Database management (stats / list / clean / export)
lint	Health check (broken links, orphans, contradictions)
doctor	System health check (config, deps, LLM, wiki, permissions, server)
sink-status	Show query sink buffer status

Auth

Command	Description
auth	Auth bootstrap (init / create-token / list-tokens / revoke / whoami)

QMD Hybrid Search

Command	Description
qmd	QMD search engine (status / search / install / embed / mcp)

MCP tools: 26 wiki_* tools available when server runs with --transport stdio or HTTP. See docs/MCP_SETUP.md.

---

Python API

from llmwikify import create_wiki

# Create or open a wiki
wiki = create_wiki("./my-wiki")

# Write a page
wiki.write_page("Python/Singleton", "# Singleton Pattern\nEnsures one instance...")

# Read a page
content = wiki.read_page("Python/Singleton")

# Search
results = wiki.search("singleton", limit=10)

# Inbound/outbound links
inbound = wiki.get_inbound_links("Python/Singleton")
outbound = wiki.get_outbound_links("Python/Singleton")

# Status / lint
status = wiki.status()
lint_result = wiki.lint()

wiki.close()

Run the unified server programmatically

from llmwikify import Wiki
from llmwikify.interfaces.server import WikiServer

wiki = Wiki("./my-wiki")
server = WikiServer(
    wiki,
    api_key="optional-secret",
    enable_mcp=True,
    enable_rest=True,
    enable_webui=True,
)
server.run(host="0.0.0.0", port=8765)

---

MCP Server (26 Tools)

Wiki maintenance and query:

Tool	Description
wiki_init	Initialize wiki structure
wiki_ingest	Ingest a source file
wiki_write_page	Write/update a wiki page
wiki_read_page	Read a wiki page
wiki_search	Full-text search (FTS5)
wiki_lint	Health check
wiki_status	Status overview
wiki_references	Page references
wiki_synthesize	Save query answer as wiki page
wiki_graph	Graph query / modify
wiki_graph_analyze	Graph export / detect / report

Multi-wiki management:

Tool	Description
wiki_list	List all registered wikis
wiki_switch	Switch to a different wiki
wiki_register	Register a new wiki
wiki_search_cross	Search across multiple wikis
wiki_scan	Scan directories for wikis

---

Documentation

• docs/TUTORIAL.md — 5 end-to-end scenarios (must read)
• docs/ONBOARDING.md — New-user guide + onboarding improvements (v0.38+)
• examples/ — 8 runnable playbooks
• ARCHITECTURE.md — Layered architecture, modules, data flow
• docs/CONFIGURATION_GUIDE.md — All config options + Doctor
• CONTRIBUTING.md — Development setup and workflow

---

Contributing

Contributions welcome! See CONTRIBUTING.md for development setup, coding standards, and the contribution workflow.

---

Acknowledgments

• llm-wiki-kit — Original inspiration
• Andrej Karpathy — LLM Wiki Principles
• Obsidian — Markdown wiki platform
• MCP — Model Context Protocol

---

License

MIT License — see LICENSE.

Contact

• GitHub: @sn0wfree
• Email: linlu1234567@sina.com
• Discussions: GitHub Discussions