llm-wiki-tools 0.4.1

CLI tools to read and manage LLM Wiki projects on GitHub

llm-wiki-tools

CLI tools for the Agentic LLM Wiki pattern. Manage project knowledge as a curated, traversal-organized Markdown wiki that AI agents can read, write, and patch surgically — entirely on your local disk.

No MCP server. No remote backend. Just Python, Markdown, and Git.

---

The concept: knowledge as code

Inspired by Andrej Karpathy's LLM Wiki pattern, this package treats project documentation like software:

1. Raw sources (raw/) — immutable "source code": PDFs, docs, transcripts, emails. The wiki is built from these.
2. Updates (updates/) — staging area for new sources awaiting ingestion. After processing, files are archived into raw/ automatically.
3. The wiki (wiki/) — "compiled binary": traversal concept pages in plain Markdown, optimized for LLM consumption. Versioned with Git.
4. The agents (Claude Code) — the "compiler". An analyst plans the structure, an ingest-agent materializes it; both invoke the CLIs below.

Pages in the wiki are concept-oriented, not source-oriented: a single source contributes to N pages, and a single page aggregates contributions from N sources. There is no one-to-one mapping between raw files and wiki pages. External traceability is preserved with inline [fonte: <slug>] markers, not with source-pages or metadata blocks.

---

How it works

       local disk (strict local — no remote backend)
       ────────────────────────────────────────────────

       wiki-my-project/
         updates/      ←  you drop new sources here
            │
            │  wiki-manage update  (reads, prints, then archives)
            ▼
         raw/          ←  immutable history; CLI-managed only
            │
            │  wiki-manage generate / update  (emits content)
            ▼
         analyst → plan → ingest-agent → wiki-manage write/patch/delete
            │
            ▼
         wiki/         ←  plain Markdown, concept pages
            │
            │  git add / commit / push  (you, manually)
            ▼
         GitHub        ←  storage and history, optional

Two CLIs:

• wiki-read — read pages and search the wiki. Used by every agent and by you.
• wiki-manage — ingest sources, write/patch/delete pages, run health checks. Used by the agents in the wiki repo.

Both operate strictly on the local filesystem. Pushing to GitHub is your responsibility, done manually with git. GitHub is just storage — the package never talks to it.

---

Install

pip install llm-wiki-tools

For OCR and image understanding (PDFs with scanned text, image files), enable the vision extra:

pip install "llm-wiki-tools[vision]"

The vision extra uses Claude as the OCR backend via the official anthropic SDK.

---

Requirements

• Python 3.12+
• Anthropic API key (optional, vision only) — set as ANTHROPIC_API_KEY or in .project to enable OCR of PDFs and images.

No GitHub token is needed by the CLI itself: Git operations are performed by you outside the package.

---

Configuration (.project)

Place a .project file at the root of your wiki repo. The CLIs walk up from the current working directory to find it.

{
  "project_id": "my-project",
  "wiki_repo": "username/wiki-my-project",
  "wiki_local_path": "/absolute/path/to/wiki-my-project",
  "anthropic_key": "sk-ant-...",
  "anthropic_model": "claude-3-5-sonnet-latest"
}

Required:

• wiki_local_path — absolute path to the wiki repo on disk. Must exist.

Optional:

• project_id, wiki_repo — identifiers used in informational output; not used for any network call.
• anthropic_key — enables vision-based OCR during raw/updates conversion.
• anthropic_model — defaults to claude-3-5-sonnet-latest.

---

Directory layout (per wiki repo)

wiki-my-project/
.project
raw/         ←  immutable archive of all ingested sources
updates/     ←  drop zone for new sources awaiting ingest
wiki/        ←  the compiled Markdown wiki (versioned with Git)

• raw/ is read-only for agents and humans. It is written exclusively by wiki-manage update as part of the archival step (see below).
• updates/ is a transit zone. You drop new files there; the CLI moves them into raw/ once successfully read.
• wiki/ is the only directory you push to GitHub.

---

Commands

wiki-read — consult the wiki

Read-only. Operates on wiki/ on the local disk.

wiki-read <page>                  # print page content
wiki-read --list                  # list all pages
wiki-read --search "<query>"      # full-text search
wiki-read --info                  # show project info from .project

Page names can omit .md. wiki-read index and wiki-read index.md are equivalent.

wiki-manage — maintain the wiki

Mixed read/write. Operates on the local disk.

Status and health

wiki-manage status                # file counts for raw/, updates/, wiki/
wiki-manage lint                  # broken links, orphan pages, empty sections, missing required

Ingestion

wiki-manage generate              # read all raw/ files, emit content for the analyst (read-only)
wiki-manage update                # read all updates/ files, emit content, then archive them into raw/

generate is purely read-only. update archives the successfully-read files into raw/ after emission. Files that fail to read remain in updates/ for a retry.

Surgical writing (via STDIN)

cat << 'EOF' | wiki-manage write <page>
# Page Title
Content...
EOF

cat << 'EOF' | wiki-manage patch <page> "## Section Header" <action>
New content...
EOF

Patch actions:

• append — insert at the end of the section
• prepend — insert immediately after the header
• replace — replace the section body
• delete — remove the section (header included)

Page removal

wiki-manage delete <page>

Refuses to delete required pages (index.md, changelog.md).

---

File lifecycle

[ you ]  →  updates/  →  [ wiki-manage update ]  →  raw/  (archive)
│
└─►  emitted content  →  analyst  →  plan
│
▼
ingest-agent
│
▼
wiki-manage write/patch
│
▼
wiki/
│
└─►  git push  (you)

Important properties:

• Archival of updates/ → raw/ is automatic and irreversible (no backup; relies on Git history for audit).
• Files that fail conversion stay in updates/. The CLI never silently drops content.
• When a file in updates/ has the same name as one in raw/, the emitted output tags it [REPLACEMENT] so the analyst knows to treat it as a revision (and to remove content that was present in the old version but absent in the new).

---

Supported source formats

The package uses markitdown to convert binary formats into Markdown.

• Plain text (read as-is): .txt, .md, .json, .jsonl, .csv, .py, .js, .ts, .go, .rs, .sql, .sh, and other common code/text extensions.
• Documents (converted via markitdown): .docx, .xlsx, .xls, .pptx, .pdf, .msg, .ipynb, .epub, .html, .xml, .zip.
• Images (described via Claude if anthropic_key is set): .jpg, .jpeg, .png, .gif, .webp. Without a key, images yield a placeholder.

Unsupported binary formats raise a clear error; the file stays in updates/.

---

The agentic system around these CLIs

This package is the execution layer. The intended usage is inside a wiki repo configured with Claude Code, where two sub-agents and a small set of skills orchestrate the CLIs:

• analyst (Opus) — reads .project, raw/, updates/, and the existing wiki; designs the page structure following anti-drift and granularity rules; produces an execution-ready plan with explicit CREATE / PATCH / DELETE primitives. Read-only.
• ingest-agent (Sonnet) — receives the plan and executes it via wiki-manage write/patch/delete, runs wiki-manage lint, writes the changelog entry. No architectural decisions.
• Skills — wiki-generate, wiki-update, wiki-lint, wiki-query. Pre-flight, delegate to sub-agents, validate the plan, present the result. Never execute writes directly.

A reference scaffolding repo (scaffolding-wiki) provides the matching CLAUDE.md, agent prompts, and skill files. This package supplies only the CLIs they depend on.

---

Wiki conventions

These are not enforced by the CLI — they are enforced by the agentic system. Two are worth stating here because they shape how wiki-manage is used:

• Pure semantic division — pages are traversal concepts (e.g. architecture.md, requirements.md, auth.md), not one-page-per-source. Never create a sources/ directory.
• Plain Markdown + inline citations — no YAML frontmatter, no JSON metadata. External provenance is tracked with inline [fonte: <slug>] markers, where <slug> is the basename of the source file (no extension, lowercased, spaces/underscores/punctuation collapsed into single dashes).

Two pages are always required:

• index.md — map of the wiki, with one description line per page.
• changelog.md — reverse-chronological log of every operation.

---

Testing locally

# install in editable mode
pip install -e ".[dev]"

# set up a fake wiki
mkdir -p /tmp/test-wiki/{raw,updates,wiki}
echo "some content" > /tmp/test-wiki/raw/doc.txt

cd /tmp
cat > .project << 'EOF'
{
  "project_id": "test",
  "wiki_repo": "username/wiki-test",
  "wiki_local_path": "/tmp/test-wiki"
}
EOF

wiki-read --info
wiki-manage status
wiki-manage generate

# run the test suite
cd /path/to/llm-wiki-tools
pytest -q

# cleanup
rm -rf /tmp/test-wiki /tmp/.project

---

Publishing to PyPI

1. Bump version in pyproject.toml (patch for fixes/subcommands, minor for new binaries, major for breaking changes to .project or CLI).
2. Update this README and CLAUDE.md if the user-facing interface or architecture changed.
3. Build and upload:

pip install build twine
python -m build
twine upload dist/*