wiki2okf 0.2.0

Convert Wikipedia articles into Open Knowledge Format bundles.

wiki2okf

Convert one Wikipedia article into a small, local, agent-ready Open Knowledge Format (OKF) v0.2 bundle.

wiki2okf convert \
  "https://fr.wikipedia.org/wiki/Alan_Turing" \
  --output ./alan-turing

alan-turing/
├── index.md
├── log.md
└── concepts/
    └── alan-turing.md

wiki2okf provides two paths:

• a fully deterministic converter with no LLM dependency, account, or API key;
• optional provider-agnostic enrichment for compact, structured knowledge.

Both paths keep the final YAML and Markdown generation inside wiki2okf. An LLM can return validated data, but it never writes the OKF files directly.

Why?

Wikipedia contains richly linked public knowledge, but full articles are often too large and noisy for an agent context. OKF provides a portable, Markdown-based way to represent concepts, sources, and relationships.

wiki2okf bridges both formats through an inspectable pipeline:

Wikipedia URL
    ↓
MediaWiki API
    ↓
cleaned WikipediaArticle
    ↓
optional two-pass enrichment
    ↓
validated models
    ↓
deterministic OKF renderer

Installation

wiki2okf requires Python 3.11 or newer.

With uv

Once the first PyPI release is published, add the deterministic package to a project:

uv add wiki2okf
uv run wiki2okf --help

Include optional LLM enrichment:

uv add "wiki2okf[llm]"

Install it as a standalone command:

uv tool install wiki2okf

Or run it without a permanent installation:

uvx wiki2okf convert \
  "https://fr.wikipedia.org/wiki/Alan_Turing" \
  --output ./alan-turing

Until the first PyPI release is available, install the current GitHub version:

uv tool install \
  "wiki2okf @ git+https://github.com/LucasMarchnd/wiki2okf.git"

With pip

pip install wiki2okf

For enrichment:

pip install "wiki2okf[llm]"

The base installation intentionally excludes LiteLLM and provider SDKs.

Deterministic conversion

The default command uses no model and follows the same inspectable conversion rules for identical Wikipedia content:

wiki2okf convert \
  "https://en.wikipedia.org/wiki/Alan_Turing" \
  --output ./alan-turing

It retrieves the article through MediaWiki, cleans the HTML, extracts categories and internal links, and renders the result without an LLM.

Content modes

concise is the default:

wiki2okf convert URL --content-mode concise

It keeps the introduction and main encyclopedic sections, excludes references, bibliographies, media lists, portals, and navigation, and limits the article body to 12,000 characters with an explicit truncation notice.

full retains more meaningful source sections while still removing navigation and portal noise:

wiki2okf convert URL --content-mode full

Optional LLM enrichment

Install the llm extra, then select any model identifier supported by LiteLLM:

wiki2okf convert \
  "https://fr.wikipedia.org/wiki/Alan_Turing" \
  --enrich \
  --model "gemini/gemini-3.1-flash-lite" \
  --fallback fail \
  --output ./alan-turing

Enrichment uses two passes:

1. Every cleaned section is assessed independently. The model assigns a relevance score and extracts up to eight grounded points.
2. The introduction and retained points are synthesized into a concise concept with a description, facts, typed relations, aliases, related concepts, and useful questions.

Every response is validated with Pydantic. Every fact and relation must name the exact Wikipedia sections supporting it. Unsupported section references, invalid limits, inconsistent relevance decisions, and malformed responses are rejected.

Providers

wiki2okf does not import provider-specific SDKs in its domain or rendering layers. LiteLLM handles provider routing.

Provider	Example model	Authentication
Google AI Studio	gemini/gemini-3.1-flash-lite	GEMINI_API_KEY
OpenAI	openai/<model>	OPENAI_API_KEY
Anthropic	anthropic/<model>	ANTHROPIC_API_KEY
Ollama	ollama_chat/<model>	None

Google AI Studio example:

export GEMINI_API_KEY="..."

wiki2okf convert URL \
  --enrich \
  --model "gemini/gemini-3.1-flash-lite"

Local Ollama example:

ollama pull qwen3.5:4b

wiki2okf convert URL \
  --enrich \
  --model "ollama_chat/qwen3.5:4b"

Small local models can struggle with strict schemas and grounding. Use --fallback fail while evaluating a model so failures are visible.

Cache and fallback

Validated responses are cached by:

• Wikipedia revision;
• provider/model identifier;
• prompt and schema versions;
• content hash;
• pipeline stage;
• temperature.

wiki2okf convert URL \
  --enrich \
  --model "gemini/gemini-3.1-flash-lite" \
  --cache-dir ~/.cache/wiki2okf

Cache entries contain validated structured data, never raw prompts or raw API responses.

The default fallback still produces the deterministic bundle if enrichment fails:

wiki2okf convert URL --enrich --model MODEL \
  --fallback deterministic

For CI, benchmarking, or model evaluation:

wiki2okf convert URL --enrich --model MODEL \
  --fallback fail

Use --keep-raw to append the cleaned source sections after the enriched concept.

Generated concept

An enriched concept remains ordinary Markdown with safe YAML frontmatter:

---
type: Mathématicien et cryptologue
title: Alan Turing
description: Alan Turing est un mathématicien et cryptologue britannique…
resource: https://fr.wikipedia.org/wiki/Alan_Turing
generated:
  by: wiki2okf/0.2.0
  at: 2026-07-24T23:00:05Z
  enrichment:
    provider: litellm
    model: gemini/gemini-3.1-flash-lite
    prompt_version: 2026-07-25.v2
    schema_version: "2"
    temperature: 0.0
sources:
  - id: wikipedia-page
    resource: https://fr.wikipedia.org/wiki/Alan_Turing
    title: Alan Turing — Wikipédia
    revision_id: 236602045
---

# Alan Turing

## Faits clés

- Alan Turing est né à Londres le 23 juin 1912. (Source: Biographie)

## Relations

- Alan Turing — a travaillé à → Bletchley Park (Source: Biographie)

The bundle also contains:

• index.md for navigation;
• log.md for import history;
• up to 20 related Wikipedia links;
• source URL, revision identifier, retrieval time, and generator provenance.

CLI reference

wiki2okf convert URL [OPTIONS]

--output, -o PATH
--content-mode concise|full
--enrich
--model PROVIDER/MODEL
--temperature FLOAT
--fallback deterministic|fail
--keep-raw
--cache-dir PATH

Current scope

Included:

• one French or English Wikipedia URL;
• MediaWiki action=parse;
• introduction and readable h2/h3 sections;
• categories and internal article links;
• concise and full content modes;
• deterministic Markdown and YAML;
• optional two-pass LLM enrichment;
• Pydantic validation, one JSON repair attempt, caching, and fallback;
• local index.md, log.md, and concept files.

Intentionally not included:

• recursive crawling;
• downloading linked pages;
• Wikidata and structured infoboxes;
• embeddings or a vector database;
• graph visualization;
• a web interface;
• bulk Wikipedia dump processing.

Development

git clone https://github.com/LucasMarchnd/wiki2okf.git
cd wiki2okf

python -m venv .venv
source .venv/bin/activate
pip install -e ".[dev,llm]"

ruff check .
pytest
python -m build

Unit tests use fake LLM implementations and never call external model APIs.

Publishing to PyPI

The repository includes a Trusted Publisher workflow in .github/workflows/publish.yml. It builds and checks the wheel and source distribution, then publishes with GitHub OIDC. No long-lived PyPI token is stored in GitHub.

For the first release, configure a pending publisher on PyPI with:

PyPI project: wiki2okf
GitHub owner: LucasMarchnd
Repository: wiki2okf
Workflow: publish.yml
Environment: pypi

Then publish a GitHub Release for the matching version. The workflow uploads the package, after which these commands become available:

uv add wiki2okf
uv tool install wiki2okf
pip install wiki2okf

Roadmap

• depth-1 crawling and local OKF links;
• Wikidata identifiers and structured infoboxes;
• incremental updates using revision_id;
• official OKF conformance validation;
• graph visualization;
• bulk Wikipedia dump processing;
• stable Python API and source plugins.

Attribution and independence

Generated bundles preserve the Wikipedia source URL and revision. Wikipedia content remains subject to the licensing and attribution requirements stated by Wikimedia and on the source article.

wiki2okf is independent and is not affiliated with or endorsed by Google, Google Cloud, the Wikimedia Foundation, Wikipedia, or any LLM provider.

License

wiki2okf is released under the MIT License.