llmwikify

Knowledge + Chat + Research Assistant — LLM-maintained knowledge bases with chat & general research capabilities.

llmwikify is a Python CLI + library + unified server for building persistent, LLM-maintained knowledge bases with chat and general research assistant capabilities.

v0.40 Refocus — quant research pipeline (paper → factor → backtest) has moved to quantnodes.research (>=4.0). llmwikify now focuses exclusively on wiki + chat + research-assistant use cases.

⚠️ v0.40.2 BREAKING for HTTP clients: POST /page to an existing page now returns 409 Conflict with a confirmation_id (one-shot, TTL 300s) instead of silently overwriting. Re-POST with the token (query string ?confirm_token=<id> is the primary path; body {"confirm_token": "..."} is the back-compat shim) to confirm. See docs/releases/v0.40.2.md for details.

⚠️ Beta Release — APIs may shift between minor versions. Report issues on GitHub.

---

Why llmwikify?

📚 Knowledge Base	LLM-maintained wiki with auto-linting, multi-wiki registry, section anchors
💬 Chat Agent	Streaming chat with tool calling, 50+ skills, subagents, memory
🔬 Research Assistant	Adaptive ReAct loop, multi-source synthesis, quality gates
🔍 Smart Search	SQLite FTS5 + optional QMD hybrid (BM25 + vector + LLM reranking)
🔗 Bidirectional Links	Automatic [[wikilink]] detection with section-level granularity
🧠 Knowledge Graph	8 relation types, PageRank, community detection, interactive D3.js visualization
🌐 Unified Server	MCP + REST + WebSocket + Web UI in one process

---

Features

💬 Chat + Skills Agent

Streaming chat with tool calling, confirmations, and 50+ built-in skills (CRUD, research, subagent, scheduler). The agent can search your wiki, manage pages, and orchestrate multi-step research — all with human-in-the-loop confirmations.

🔬 Research Assistant

Adaptive ReAct research engine that breaks down questions into sub-queries, gathers sources, applies quality gates, and synthesizes structured reports. Configuration-driven via ~/.llmwikify/llmwikify.json.

📚 Knowledge Base (Wiki)

LLM-maintained wiki with bidirectional wikilinks, multi-wiki registry, section-level anchors, YAML frontmatter + DuckDB storage, FTS5 full-text search.

Markdown Editor

Split-pane live markdown editor with page tree, front-matter panel, and wikilink autocomplete. Edit, preview, and manage your wiki pages in one view.

Knowledge Graph

Interactive D3.js force-directed graph with PageRank node sizing, community coloring, and bridge highlighting. Explore relationships between your wiki pages visually.

Dashboard

Track your knowledge growth with metrics cards, Wiki Dream activity timeline, and health indicators. See how your wiki evolves over time.

---

Quick Start

# Install (with PDF/document extractors + web server)
pip install 'llmwikify[extractors,web]'

# Create and enter wiki directory
mkdir my-wiki
cd my-wiki

# Initialize a wiki
llmwikify init

# Add a page manually
llmwikify write_page "hello" --content "# Hello World\n\nThis is my first page."

# Ingest a markdown source (PDF requires [extractors])
llmwikify ingest README.md

# Build the link index (finds [[wikilinks]])
llmwikify build-index

# Search the wiki
llmwikify search "hello"

# Read a page
llmwikify read_page hello

# Show wiki status
llmwikify status

# Start the server (MCP + REST + Web UI)
llmwikify serve --web --port 8765
# Open http://localhost:8765

LLM Setup (for AI features)

# Option 1: One-shot — set up LLM with your wiki
export OPENAI_API_KEY=sk-...
llmwikify init --llm

# Option 2: Standalone — set up LLM separately
llmwikify init-llm
llmwikify init-llm --provider openai --api-key sk-...
llmwikify init-llm --provider anthropic  # uses ANTHROPIC_API_KEY
llmwikify init-llm --provider minimax    # uses MINIMAX_API_KEY

# Option 3: Interactive — init will prompt you
llmwikify init
# 💡 LLM features (analyze-source, synthesize, chat) need ~/.llmwikify/llmwikify.json
#    No LLM config detected. Set one up now? [y/N]: y
#    Detected OPENAI_API_KEY in env vars. Provider: openai
# ✅ LLM config written to ~/.llmwikify/llmwikify.json

Custom endpoint / OpenAI-compatible

llmwikify init-llm \
    --provider openai \
    --api-key sk-... \
    --base-url https://api.deepseek.com/v1

Non-interactive (CI / scripts)

llmwikify init --llm --no-llm-prompt --llm-overwrite

Without LLM: init, search, write_page, read_page, build-index, references, lint, graph-analyze, export-graph all work offline.

---

Doctor — Health Check

llmwikify doctor                     # Check everything (5s LLM test)
llmwikify doctor --skip-llm          # Skip LLM API call (faster)
llmwikify doctor --wiki-root /path   # Check a specific wiki
llmwikify doctor --json              # JSON output for CI/scripts

What it checks

#	Check	What
1	Config	~/.llmwikify/llmwikify.json exists, parseable, has valid api_key
2	Python	Version >= 3.10
3	Core deps	llmwikify, yaml, duckdb, jinja2
4	Optional extras	fastapi, fastmcp, watchdog, networkx, markitdown, tiktoken, httpx
5	LLM connectivity	Actually calls provider API (5s timeout) with "Say hi" — verifies key works
6	Wiki directory	4 functional paths (raw/, wiki/, .llmwikify.db, wiki.md) + all page-type subdirs declared in wiki.md (parsed dynamically from Directory Structure + Page Types sections). See docs/DOCTOR.md
7	Permissions	~/.llmwikify/ and wiki root are writable
8	WebUI bundle	ui/webui/dist/index.html exists
9	Server	GET /api/health returns 200

Severity and fix

Every fail or actionable warn carries a fix dict (commands + docs + cost + risk + auto). In text mode these are aggregated at the bottom under 📋 Recommended actions:

📋 Recommended actions:

  [FAIL 1/1] wiki (4 functional paths missing)
      Fix:   llmwikify init
      Docs:  docs/ONBOARDING.md#init
      Cost:  ~5s    Risk: low    Auto: True

In JSON mode (--json), each check has a fix field on actionable items. Severity: FAIL blocks exit 0; WARN and INFO do not.

Exit codes

Code	Meaning
0	All checks passed (warnings/info allowed)
1	One or more checks failed
2	Config missing — run llmwikify init-llm

Examples

# CI integration (JSON + skip LLM)
llmwikify doctor --json --skip-llm | jq -e '.summary.failed == 0'

# Quick check before running expensive operations
llmwikify doctor --skip-llm  # ~1 second

# Check a specific wiki
llmwikify doctor --wiki-root /home/ll/Public/comovement

For the full reference (all 9 checks, fix dict schema, troubleshooting, CI recipes), see docs/DOCTOR.md.

---

Tutorial

New to llmwikify? Start with our 5 end-to-end scenarios:

#	Scenario	Description
1	Personal Reading Notes	PDF → searchable wiki with cross-references
2	Company Due-Diligence KB	Multi-source analysis → knowledge graph
3	Multi-Wiki Collaboration	Manage multiple wikis through one server
4	Chat + ReAct Agent	LLM-powered Q&A with tool calling
5	Quant Reproduction	Paper → Factor → Backtest → L5 reflection

📖 Full tutorial: docs/TUTORIAL.md (40-60 min read) 🎯 Runnable examples: examples/ (8 playbooks, no LLM required)

---

Features at a Glance

Feature	Description
Wiki Core	FTS5 search, bidirectional references, query compounding, multi-wiki registry
Smart Lint	Broken links, orphans, contradictions, outdated pages, knowledge gaps
Knowledge Graph	8 relation types, PageRank, community detection, HTML/SVG/GraphML export
Chat + Agent	ReAct streaming, 26 MCP tools, skills system, research engine
Research Assistant	Adaptive ReAct loop, multi-source synthesis, quality gates (general purpose)
Web UI	React SPA: editor, graph, dashboard, chat pages
Extraction	PDF, Word, Excel, PowerPoint, images, audio, web, YouTube
MCP Server	26 tools over stdio + HTTP, multi-wiki support

---

Architecture

graph TB
    subgraph "Interfaces"
        CLI[CLI]
        MCP[MCP Server]
        REST[REST API]
        UI[Web UI]
    end
    
    subgraph "Apps"
        Wiki[Wiki Service]
        Chat[Chat + ReAct]
        Research[Research Engine]
        Agent[Agent Runtime]
    end
    
    subgraph "Kernel"
        Engine[Wiki Engine]
        Graph[Knowledge Graph]
        Search[Search Engine]
        Storage[SQLite Storage]
    end
    
    subgraph "Foundation"
        LLM[LLM Client]
        Extract[Extractors]
        Config[Configuration]
    end
    
    subgraph "Reproduction"
        Paper[Paper Pipeline]
        Factor[Factor Library]
        Backtest[Backtest Engine]
    end
    
    CLI --> Wiki
    MCP --> Wiki
    REST --> Wiki
    UI --> Chat
    Chat --> Engine
    Research --> LLM

v0.40 architecture: strict 4-layer foundation → kernel → apps → interfaces with single-direction dependencies. The quant module (paper / factor / backtest / strategy) was stripped and migrated to the sibling project quantnodes. See docs/REFACTORING.md for the full refactor history, dependency matrix, and cross-project migration.

---

Installation

pip install llmwikify              # Core (zero hard deps)
pip install llmwikify[all]         # Full features
pip install llmwikify[web]         # Web UI + REST
pip install llmwikify[mcp]         # MCP server
pip install llmwikify[extractors]  # PDF/Office/media

Optional Extras

Extra	Purpose
extractors	PDF / Office / images / audio / YouTube via MarkItDown
mcp	MCP server (fastmcp)
watch	Filesystem watching (watchdog)
graph	Graph visualization + community detection
web	FastAPI / Starlette / Uvicorn for the unified server
agent	Scheduler + filelock + DuckDuckGo / Tavily search
llm	tiktoken for token counting
all	Everything above

---

CLI Reference

Core Wiki Operations

Command	Description
init	Initialize a wiki (creates dirs + wiki.md schema + .llmwikify.db)
init --llm	Initialize wiki + set up LLM config (one-shot)
init-llm	Set up LLM config only (auto-detects from OPENAI_API_KEY etc.)
write_page	Write/update a wiki page (llmwikify write_page "Name" --content "...")
read_page	Read a wiki page (name only, no .md suffix)
ingest	Ingest a source file or URL (PDF requires [extractors])
batch	Batch ingest a directory of sources
status	Show wiki stats (page count, index, links)
log	Record an operation in log.md

Search & Analysis

Command	Description
search	Full-text search (FTS5 backend)
analyze-source	LLM-powered source analysis and caching
knowledge-gaps	Detect missing pages, outdated content, redundancy
suggest-synthesis	LLM: generate cross-source synthesis suggestions
synthesize	Save query answer as a wiki page
report	Generate unexpected connections report

Link Index & References

Command	Description
build-index	Build bidirectional link index from [[wikilinks]]
references	Show inbound/outbound links, broken links, stats
fix-wikilinks	Fix broken wikilinks by adding directory prefix

Knowledge Graph

Command	Description
graph-analyze	PageRank, community detection, suggestions
graph-query	Query graph (neighbors / path / stats / context)
community-detect	Detect knowledge communities (Leiden/Louvain)
export-graph	Export visualization (HTML/SVG/GraphML)

Multi-Wiki

Command	Description
wikis	Multi-wiki management (list / add / remove / scan)

Server & MCP

Command	Description
serve	Start unified server — MCP + REST + Web UI (alias: mcp)
watch	Watch raw/ directory for new files, auto-ingest

Quant Research

v0.40 BREAKING: quant-init and reproduce commands removed. Quant research pipeline moved to quantnodes.research (>=4.0). Use quantnodes reproduce instead.

Database & Health

Command	Description
db	Database management (stats / list / clean / export)
lint	Health check (broken links, orphans, contradictions)
doctor	System health check (config, deps, LLM, wiki, permissions, server)
sink-status	Show query sink buffer status

Auth

Command	Description
auth	Auth bootstrap (init / create-token / list-tokens / revoke / whoami)

QMD Hybrid Search

Command	Description
qmd	QMD search engine (status / search / install / embed / mcp)

MCP tools: 26 wiki_* tools available when server runs with --transport stdio or HTTP. See docs/MCP_SETUP.md.

---

Python API

from llmwikify import create_wiki

# Create or open a wiki
wiki = create_wiki("./my-wiki")

# Write a page
wiki.write_page("Python/Singleton", "# Singleton Pattern\nEnsures one instance...")

# Read a page
content = wiki.read_page("Python/Singleton")

# Search
results = wiki.search("singleton", limit=10)

# Inbound/outbound links
inbound = wiki.get_inbound_links("Python/Singleton")
outbound = wiki.get_outbound_links("Python/Singleton")

# Status / lint
status = wiki.status()
lint_result = wiki.lint()

wiki.close()

Run the unified server programmatically

from llmwikify import Wiki
from llmwikify.interfaces.server import WikiServer

wiki = Wiki("./my-wiki")
server = WikiServer(
    wiki,
    api_key="optional-secret",
    enable_mcp=True,
    enable_rest=True,
    enable_webui=True,
)
server.run(host="0.0.0.0", port=8765)

---

MCP Server (26 Tools)

Wiki maintenance and query:

Tool	Description
wiki_init	Initialize wiki structure
wiki_ingest	Ingest a source file
wiki_write_page	Write/update a wiki page
wiki_read_page	Read a wiki page
wiki_search	Full-text search (FTS5)
wiki_lint	Health check
wiki_status	Status overview
wiki_references	Page references
wiki_synthesize	Save query answer as wiki page
wiki_graph	Graph query / modify
wiki_graph_analyze	Graph export / detect / report

Multi-wiki management:

Tool	Description
wiki_list	List all registered wikis
wiki_switch	Switch to a different wiki
wiki_register	Register a new wiki
wiki_search_cross	Search across multiple wikis
wiki_scan	Scan directories for wikis

---

Documentation

• docs/TUTORIAL.md — 5 end-to-end scenarios (must read)
• docs/ONBOARDING.md — New-user guide + onboarding improvements (v0.38+)
• examples/ — 8 runnable playbooks
• ARCHITECTURE.md — Layered architecture, modules, data flow
• docs/CONFIGURATION_GUIDE.md — All config options + Doctor
• docs/DOCTOR.md — llmwikify doctor reference: 9 checks, fix dicts, CI integration (v0.40+)
• CONTRIBUTING.md — Development setup and workflow

---

Contributing

Contributions welcome! See CONTRIBUTING.md for development setup, coding standards, and the contribution workflow.

---

Acknowledgments

• llm-wiki-kit — Original inspiration
• Andrej Karpathy — LLM Wiki Principles
• Obsidian — Markdown wiki platform
• MCP — Model Context Protocol

---

License

MIT License — see LICENSE.

Contact

• GitHub: @sn0wfree
• Email: linlu1234567@sina.com
• Discussions: GitHub Discussions