wiki-forest 0.1.0

Forest.Wikipedia.EN: a populated English Wikipedia terrain with sealed semantic retrieval.

Forest.Wikipedia.EN

The English Wikipedia forest: populated, local, and ready for people and agents.

Forest.Wikipedia.EN turns an offline English Wikipedia canopy into stable article bearings, authored sections, exact passages, and bounded semantic search. It is designed to answer where should I look? without pretending that retrieval alone makes something true.

The populated release contains 8,419,072 canonical article bearings and a sealed semantic index. Search and canopy navigation work locally. Exact article depth is fetched from Wikimedia only when a user or agent deliberately enters an article, then cached by revision and content hash.

Canopy search gives directions, not evidence. A consuming application decides what was exposed, why it was selected, and what authority it carries.

Get the populated Forest

The code lives on GitHub. The large, versioned Wikipedia canopy lives on Hugging Face.

pip install "wiki-forest[semantic]"
wiki-forest setup
wiki-forest search "milk"

setup downloads the pinned populated release, resumes interrupted transfers, verifies every artifact, and creates a fresh local database. Allow about 20 GB of free disk space.

Until the package is published, install from a checkout:

git clone https://github.com/schmerbert/Forest.Wikipedia.EN.git
cd Forest.Wikipedia.EN
python -m venv .venv
# Windows: .venv\Scripts\activate
# Unix:    source .venv/bin/activate
pip install -e ".[semantic,test]"
wiki-forest --root . identity

What is populated?

Layer	Contents	Approximate size
Canopy	Kiwix English Wikipedia all_mini, June 2026	11.67 GiB
Semantic projection	FAISS IVF-PQ index plus SQ8 reranking codes	3.47 GiB
Canonical inventory	Stable mapping for 8.4 million bearings	0.78 GiB
Query model	Pinned BAAI/bge-small-en-v1.5 snapshot	0.13 GiB

Personal runtime state is never part of the populated release. Hydrated article bodies live under sources/, while traversal metadata and revision pins live in forest.db; both begin empty on every new installation.

Try it

Resolve an exact bearing without entering the article:

wiki-forest bearing "Milk"

Search the unread canopy:

wiki-forest search "rituals around hospitality" --limit 8

Inspect the mounted snapshot and semantic identity:

wiki-forest identity
wiki-forest status
wiki-forest verify --quick

Hydrate exact depth on first entry:

wiki-forest hydrate "Milk"
wiki-forest children "Milk"
wiki-forest node "Milk" "History.p1"

Full verification reads and hashes the entire populated release and can take a while:

wiki-forest verify

Remove the downloaded canopy, indexes, local database, and hydrated articles:

wiki-forest remove-data --dry-run
wiki-forest remove-data
pip uninstall wiki-forest

The command reports the exact location and size, requires confirmation, and preserves files it does not own. Use --yes only for unattended removal. It refuses to operate on a source checkout or filesystem root.

Set a custom data location with WIKI_FOREST_HOME or pass --root. Existing checkout-based integrations remain supported:

wiki-forest --root D:\data\wiki-forest identity

Offline boundary

The bundled all_mini archive contains every article's title, introduction, and infobox, without images. Consequently:

• Bearing lookup and canopy search work offline.
• The sealed semantic index works offline after installation.
• Full authored sections and passages require network access on first entry.
• A hydrated revision is pinned and reused locally; it never silently updates.

This is deliberate. Forest.Wikipedia.EN keeps a complete shallow world available while allowing exact depth to grow only where someone actually walks.

Stable addresses, readable labels

Every node has a stable machine locator. Existing addresses remain valid:

article              article root
article.p1           first lead passage
History              authored section
History.p1           first passage in History

APIs also return display_title, so interfaces can show Lead passage 1 or History — passage 1 without exposing internal locator syntax.

Architecture

canopy bearing
    → article lead
        → section map
            → section
                → subsection
                    → passage

• canopy/ holds the sealed Kiwix archive and checksum.
• sources/ holds immutable hydrated revision bodies, addressed by SHA-256.
• projections/ holds rebuildable structural and semantic indexes.
• forest.db holds source custody, revision pins, and navigable nodes.
• models/ holds the pinned query encoder used by semantic retrieval.

The production retriever unions exact and lexical ZIM bearings with FAISS candidates, reranks a bounded neighborhood from compact SQ8 codes, then applies diversity selection. Exact title matches cannot be displaced merely because the approximate index omitted them.

Long-lived applications can keep the verified index and model warm:

wiki-forest-index serve-production

The worker accepts newline-delimited JSON on standard input and emits one result per line. See DESIGN.md for the full retrieval design, benchmark history, integrity model, and rebuild procedure.

Forest ecosystem

Forest.Wikipedia.EN is usable by itself and is also a populated backend for the wider Forest ecosystem. Its catalog identity is forest.wikipedia.en. It owns source identity and navigable Wikipedia terrain. A consumer owns the reason for retrieval: actor, query, exposure, selection, interpretation, and consequences.

The dotted name is designed to scale as a catalog: forest.wikipedia.fr, forest.arxiv.cs, or forest.docs.personal can identify other populated forests without inventing a new project metaphor each time. For compatibility, the Python package and commands remain wiki-forest, the import remains wiki_forest, and the original integration field remains forest_id: wikipedia-en-lazy throughout the v0.1 line.

That boundary allows many applications to share one sealed Wikipedia substrate without confusing a retrieved source with locally adopted truth.

Development

pip install -e ".[test]"
pytest -q

Dense-index construction dependencies are intentionally optional. Build and benchmark commands live under wiki-forest-index; ordinary lexical traversal does not require FAISS or PyTorch.

Provenance and license

Forest.Wikipedia.EN code is MIT licensed. The populated release contains independently licensed Wikipedia, Kiwix/openZIM, and model material. See THIRD_PARTY_NOTICES.md before redistributing a bundle.

Forest.Wikipedia.EN is independent and is not endorsed by Wikimedia Foundation, Kiwix, openZIM, BAAI, or Hugging Face.

Status

v0.1.0 is the first public compatibility release. Locators, CLI commands, and machine-readable identity fields used by existing Forest consumers are treated as stable within the v0.1 line.