Wikifier

A zero-dependency codebase wiki for AI agents — token-efficient maps so LLMs look things up instead of re-reading full sources.

Wikifier is an agent-to-agent tool: it builds a living map of a project (health matrix, dependency graph, short file summaries) and agents keep that map current as they work. Humans can peek via a small dashboard; the product is the agent loop, not a general docs site or IDE.

Works from small scripts to large monorepos. Deep import/include maps (zero-dependency parsers):

Health/journal still work for any monitored path. The non-Python parsers are pragmatic regex (not full cargo/go.mod/classpath/-I resolution). On huge monorepos, split scope deliberately:

Or pass --directory=pkg/ / --max-files=N per run. Raise dirty cap with WIKIFIER_CHECK_CHANGES_MAX (default 2000) only when needed. Never set project_root to a multi-repo parent of clones.

Why

Context windows are finite. Re-reading a large file to answer “what is this and who depends on it?” wastes tokens.

Map first, wiki depth second: update-maps builds the structural map automatically. Rich per-file wiki text is filled by agents as they work — not a free full-repo “understand everything” pass on init.

First run (bootstrap the map)

pip install wikifier            # pure Python stdlib core — no runtime deps
pip install wikifier[mcp]       # optional Model Context Protocol (MCP) server

cd /path/to/your/project
wikifier init                   # seeds index.html + lean path-list templates
# Edit monitored_paths.txt + map_paths.txt to package roots (not bare ".") on real trees
wikifier update-maps            # full structural map → library.md + import cache
wikifier health --summary       # matrix counts
wikifier suggest-next           # or MCP suggest_next_actions — 🔴/🟡 only

Always set an explicit root for external trees: WIKIFIER_PROJECT_ROOT=/abs/path wikifier …

MCP session_bootstrap → readiness: blocked? That means lean scope and/or the map are missing (often bare . monitor + never ran update-maps) — not a broken install. Fix: write lean monitored_paths.txt / map_paths.txt, then update-maps. Agent contract: skills/run.md § Readiness.

Steady state (only touch what needs it)

Full protocol: skills/run.md (Agent Protocol v0.7 — package 4.7.x).

wikifier session-bootstrap      # one-shot: root, health, attention, actions[], names-only map_index
wikifier check-changes          # content-honest dirty; red ghosts (missing paths)
# prioritize 🔴 then *actionable* 🟡 — do NOT re-wiki 🟢 Green files
wikifier prepare-edit path/file.py   # wiki + status + deps/dependents preflight
# ... edit only those sources ...
wikifier record-change "path/file.py" "why this changed"   # required
# ... refresh that file’s wiki summary only ...
wikifier mark-green "path/file.py"   # refused if the wiki misses public symbols you added/removed
# many files at once (reasons per path, glob or dir/; git supplies the file list):
wikifier record-changes -r "src/api/=retry on 429" -r "tests/=cover retries" --only src/ --only tests/ --green
wikifier update-maps            # only if imports/structure changed (warm 0-dirty is cheap)
# removals:
wikifier record-deletion "path/gone.py" "why removed"

Core 6 (prefer every session — MCP or library/CLI):
session_bootstrap → check_changes → prepare_edit → suggest_next_actions (json actions[]) → record_change → mark_green.

Advanced intel as needed: list_paths (names-only folder expand), get_dependencies, get_dependents, get_cycles, barrels/diagnostics. Always pass project_root= / WIKIFIER_PROJECT_ROOT for external trees. Never point project_root at a multi-repo parent folder (e.g. a directory of clones).

Map split: update-maps writes two views from the same import cache — human library.md (File Tree + mermaid, dashboard only) and sharded agent folder cards (.wikifier_staging/maps/, depth-1). Bootstrap returns the root card; list_paths expands one folder; prepare_edit follows file-to-file links. Do not read library.md mermaid for orientation.

What you get

• Import analysis — Python (ast), JS/TS (ESM/CJS, barrels), Rust (use/mod + best-effort crate:: paths), Go, C/C++ includes, C# usings, Java; per-edge confidence; barrel expansion for TS/JS
• Incremental pipeline — pure-Python update-maps: dirty parse → import cache → reverse deps → cycles → library.md
• Warm agent maps (4.6.3–4.6.7) — zero-dirty + index-first candidates (re-list only when fingerprint / map-scoped index / live count disagree); MapScope keeps collect, live count, index filter, and prune aligned; stdlib SQLite; content-hash dirty
• Two path lists — map_paths.txt = map package roots; monitored_paths.txt = extra watch list (docs, scripts). check-changes always watches mapped files and anything ever marked Green, so a narrow list never hides edits
• Green you can trust — mark-green checks the wiki against the code's public symbols (Python, JS/TS) and refuses when added symbols are undocumented or removed ones are still referenced; verify-wikis audits every Green wiki; health --summary splits Green into verified / no-wiki / forced
• Partial-map honesty — map_coverage on update_maps / bootstrap / suggest_next; update_maps_until_complete when incomplete
• Cache ops — wikifier cache-status; JSON dual-write deprecated default-off (WIKIFIER_CACHE_JSON=1 opt-in); dual-read for migrate
• Selective agent work — health + suggest bias to 🔴/actionable 🟡 only; ACS v1.3 reason_code / agent_signal; prefer actionable_low_conf_edges + reason codes — never raw low_conf_edges averages alone
• Scale — indexed edge table (dependents/dependencies without loading the cache), exact barrel invalidation, nanosecond-mtime dirty checks (unchanged files are never re-read)
• MCP tools — optional server for Claude, Cursor, Cline, and other MCP clients
• Zero core dependencies — stdlib only; forks can add their own stack on top
• Honest failures — a missing project_root, missing file or lock timeout is an error result, never a silent fallback; every command is plain Python (the shell launchers just exec python -m wikifier)

Does it pay off? (measured)

scripts/benchmark_lookup.py asks the questions an agent asks before an edit, on real repositories, answered by Wikifier and by grep, scored against Python's own bytecode import scanner (details). 40 targets per repo:

Typical lookups cost about the same; grep goes wrong on common module names (base, utils) and its cost explodes on them (up to 100k tokens for one question). Mapping from scratch: 2.3 s for 724 files, 21 s for 7,174.

Performance (measured)

Full / heavy runs (historical order-of-magnitude):

Warm 0-dirty re-runs after 4.6.7 (same machine class; scoped; candidates reused — agent session path):

Residual floor on large scopes is mtime/stat + live count under MapScope (not full JSON re-walk). Sub-100ms is not a hard SLA on every 1k+ tree.

Parser accuracy

python scripts/parser_accuracy.py measures precision/recall of internal edges on small realistic layouts (tests/accuracy_fixtures.py):

Tests: python -m unittest discover tests (stdlib only; 260+ tests including one regression test per 4.7.0 fix, the accuracy fixtures and the benchmark harness).

Commands

Library: from wikifier import session_bootstrap, prepare_edit, list_paths, check_changes, record_change, record_changes, mark_green, verify_wikis, suggest_next_actions, update_maps, get_dependencies, get_dependents, cycles_report, health, init_project.

wikifier.sh / wikifier.ps1 / wikifier.bat are thin launchers for python -m wikifier (set WIKIFIER_PYTHON to pick the interpreter).

MCP

WIKIFIER_PROJECT_ROOT=/abs/path/to/project wikifier-mcp
# or: python3 -m wikifier.mcp.server

Setup and tool list: wikifier/mcp/README.md.

Human dashboard (secondary)

wikifier init drops a single index.html. Prefer wikifier serve (e.g. http://localhost:8787/index.html) — file:// can’t load project files. The markdown artifacts and CLI/MCP tools stay the source of truth; the UI is a read-only window.

wikifier serve also exposes a read-only JSON API (/__wikifier/api/file?path=…, dependencies, dependents, cycles, journal, bootstrap, diagnostics) that the dashboard uses for per-file dependencies and history. This repo also contains a proposed redesign, index.v2.html and diagnostics.v2.html, to compare side by side with the current pages (http://localhost:8787/index.v2.html); they are not deployed by init.

Scope

In: agent-maintained codebase wiki, dependency intelligence, token-saving lookup for LLMs and coding agents.
Out: general human documentation systems, IDE plugins, “docs for everyone” product growth.

Agent navigability: Prefer the protocol (skills/run.md) + MCP Core 6 over reading the parsers/cache. Self-tests live under tests/ and tests/selftest/.

Links

• PyPI · GitHub
• Agent protocol: skills/run.md
• Changelog: CHANGELOG.md
• Dogfood notes: Findings/ (historical plans and research in Findings/archive/)