Forest.Wikipedia.EN

The English Wikipedia forest: populated, local, and ready for people and agents.

Forest.Wikipedia.EN turns an offline English Wikipedia canopy into stable article bearings, authored sections, exact passages, and bounded semantic search. It is designed to answer where should I look? without pretending that retrieval alone makes something true.

The populated release contains 8,419,072 canonical article bearings and a sealed semantic index. Search and canopy navigation work locally. Exact article depth is fetched from Wikimedia only when a user or agent deliberately enters an article, then cached by revision and content hash.

Canopy search gives directions, not evidence. A consuming application decides what was exposed, why it was selected, and what authority it carries.

Get the populated Forest

pip install "wiki-forest[semantic]"
wiki-forest setup
wiki-forest search "milk"

setup downloads the pinned populated release from Hugging Face, resumes interrupted transfers, verifies every artifact, and creates a fresh local database. Allow about 20 GB of free disk space.

The first semantic search arms a local warm worker (often tens of seconds on a cold machine). Later searches reuse it. Explicit hangar controls:

wiki-forest arm
wiki-forest status
wiki-forest disarm

From a source checkout:

git clone https://github.com/schmerbert/Forest.Wikipedia.EN.git
cd Forest.Wikipedia.EN
python -m venv .venv
# Windows: .venv\Scripts\activate
# Unix:    source .venv/bin/activate
pip install -e ".[semantic,test]"
wiki-forest --root . identity

What is populated?

Personal runtime state is never part of the populated release. Hydrated article bodies live under sources/, while traversal metadata and revision pins live in forest.db; both begin empty on every new installation.

Try it

Resolve an exact bearing without entering the article:

wiki-forest bearing "Milk"

Search the unread canopy (auto-arms the warm worker on first use):

wiki-forest search "rituals around hospitality" --limit 8

Inspect readiness, mounted identity, and whether the semantic worker is armed:

wiki-forest status
wiki-forest identity
wiki-forest verify --quick

Map before you read. map returns locators and labels without passage bodies. Human section titles resolve to stored MediaWiki locators:

wiki-forest map "Milk"
wiki-forest map "Milk" "Etymology and terminology"
wiki-forest map "Milk" "Etymology_and_terminology"
wiki-forest node "Milk" "Etymology_and_terminology.p1"

children remains the full-body layer for agents that already know they want text. Both map and children hydrate on first entry unless --offline is set.

Full verification reads and hashes the entire populated release and can take a while:

wiki-forest verify

Successful setup and full verification write a local verification receipt. Ordinary searches validate that receipt using the trusted packaged manifest and artifact metadata, avoiding repeated multi-gigabyte hashing. wiki-forest verify always reads and hashes every declared byte again.

Remove the downloaded canopy, indexes, local database, and hydrated articles:

wiki-forest remove-data --dry-run
wiki-forest remove-data
pip uninstall wiki-forest

The command disarms any warm worker, reports the exact location and size, requires confirmation, and preserves files it does not own. Use --yes only for unattended removal. It refuses to operate on a source checkout or filesystem root.

Set a custom data location with WIKI_FOREST_HOME or pass --root. Existing checkout-based integrations remain supported:

wiki-forest --root D:\data\wiki-forest identity

Acceptance

Reference timings on a populated Windows checkout (2026-08-10), measured with python scripts/measure_acceptance.py. Search uses a long-lived production worker; bearing/map/node use the ordinary CLI.

These are not SLAs. Cold arm is dominated by loading the query encoder and mapping the sealed index; warm search should stay well under a second once the worker is resident. Re-run the script on your machine after wiki-forest setup.

Offline boundary

The bundled all_mini archive contains every article's title, introduction, and infobox, without images. Consequently:

• Bearing lookup and canopy search work offline.
• The sealed semantic index works offline after installation.
• Full authored sections and passages require network access on first entry.
• A hydrated revision is pinned and reused locally; it never silently updates.
• Hydrated source custody records include the Parsoid implementation and MediaWiki HTML DOM specification versions embedded in the fetched document.
• --offline refuses first-entry hydration when an article is not already pinned.

This is deliberate. Forest.Wikipedia.EN keeps a complete shallow world available while allowing exact depth to grow only where someone actually walks.

Stable addresses, readable labels

Every node has a stable machine locator. Existing addresses remain valid:

article              article root
article.p1           first lead passage
History              authored section
History.p1           first passage in History
Etymology_and_terminology     section whose MediaWiki id uses underscores

Human titles such as Etymology and terminology resolve to the stored locator. APIs also return display_title, so interfaces can show Lead passage 1 or History — passage 1 without exposing internal locator syntax.

Architecture

canopy bearing
    → article lead
        → section map
            → section
                → subsection
                    → passage

• canopy/ holds the sealed Kiwix archive and checksum.
• sources/ holds immutable hydrated revision bodies, addressed by SHA-256.
• projections/ holds rebuildable structural and semantic indexes.
• forest.db holds source custody, revision pins, and navigable nodes.
• models/ holds the pinned query encoder used by semantic retrieval.

The production retriever unions exact and lexical ZIM bearings with FAISS candidates, reranks a bounded neighborhood from compact SQ8 codes, then applies diversity selection. Exact title matches cannot be displaced merely because the approximate index omitted them.

Long-lived applications can keep the verified index and model warm:

wiki-forest arm
wiki-forest-index serve-production

wiki-forest search auto-arms a localhost NDJSON worker (preferring a checkout .index-venv when FAISS lives there). The stdin worker from wiki-forest-index serve-production remains available for embedded hosts. See DESIGN.md for retrieval design, geometry benchmarks, integrity, and rebuild notes.

Forest ecosystem

Forest.Wikipedia.EN is usable by itself and is also a populated backend for the wider Forest ecosystem. Its catalog identity is forest.wikipedia.en. It owns source identity and navigable Wikipedia terrain. A consumer owns the reason for retrieval: actor, query, exposure, selection, interpretation, and consequences.

Names

The dotted catalog scales cleanly: forest.wikipedia.fr, forest.arxiv.cs, or forest.docs.personal can name other populated terrains without a new metaphor each time.

Related — not the same

The_Forest is the resident custody memory layer (signed entries, ticketed walk, optional root). This repository is sealed wild Wikipedia terrain a host may search and cite into that store. Similarity here returns directions; it never promotes anything to ground. Crossing into a resident’s adopted truth is the consumer’s ceremony, not this package.

That boundary lets many applications share one sealed Wikipedia substrate without confusing a retrieved source with locally adopted truth.

Development

pip install -e ".[semantic,test]"
pytest -q
python scripts/measure_acceptance.py

Dense-index construction dependencies are intentionally optional. Build and benchmark commands live under wiki-forest-index. Checkout builders often keep FAISS/PyTorch in .index-venv; the warm worker prefers that interpreter when present. Ordinary lexical traversal does not require FAISS or PyTorch.

Provenance and license

Forest.Wikipedia.EN code is MIT licensed. The populated release contains independently licensed Wikipedia, Kiwix/openZIM, and model material. See THIRD_PARTY_NOTICES.md before redistributing a bundle.

Forest.Wikipedia.EN is independent and is not endorsed by Wikimedia Foundation, Kiwix, openZIM, BAAI, or Hugging Face.

Status

v0.1.1 polishes the hangar over the same populated Hugging Face artifact digests as v0.1.0 (catalog identity string updated in the bundle manifest). Locators, core CLI commands, and machine-readable search fields remain stable within the v0.1 line. Additive surfaces: map, arm, disarm, hydrated_now, worker, and Acceptance timings.