Wikiparse: Wikitext Translation Pipeline

A tool that parses wikitext into markdown+YAML from scratch in Python, including a powerful citation parser based purely on optimized regular expressions.

wikiparse converts raw Wikipedia XML dumps into structured, per-article Markdown with YAML frontmatter — from scratch, with no reliance on the MediaWiki API. It streams a dump chunk by chunk, translates the wikitext primitives (templates, tables, tags, links) into clean Markdown, and canonicalizes every {{cite ...}} / <ref> it finds into deduplicated reference records.

But... why?

I think this repository might be useful for a few different audiences:

Wikipedia

For Wikipedia engineers

No code was adapted from any of the (surely incredible) Wikimedia Foundation's repositories, due to their size and my unfamiliarity with PHP. I shudder to think of the man-hours I respent in places, but I think the result might be useful in an unexpected way: as a fresh, radically-distinct look at how someone might approach rendering & indexing wikitext corpora, especially in the context of artificial consumers. The parts most likely to travel on their own: the regex stores that route templates, split spanned table cells, and identify references (wikiparse.data.regex); the multistream runner that leases dump streams to workers on any number of hosts and commits each finished stream as one zstd row in an immutable, checksummed SQLite store; and the census that fell out of running it over all 256,564 streams of enwiki-20260601 — 59,026,190 raw {{cite}} invocations, and 13,448,738 of the 68,588,243 paired <ref> spans (19.6 %) carrying no recognizable cite template at all. Those numbers are the refs, templates, and streams tables of the dataset below, per stream, so you can check them.

For Wikipedia editors

Who want a faithful, offline, reproducible wikitext-to-Markdown conversion with real citation handling. Every {{cite}} becomes one canonical record with a stable identity (smith-jane__1999__beans), so the same source cited on a thousand pages is one thing you can count, and a citation with no author is still kept — keyed by its publisher and title rather than dropped on the floor. The articles table scores 5,135,607 cited articles by how many of their references parsed cleanly, and categories does the same across 529,402 categories: not a verdict on anyone's writing, but a whole-dump map of where the references are hardest to read, which is a fine place to point a maintenance drive.

For Wikipedia users

Who have wondered what the encyclopedia looks like from outside — as one object, rather than one page at a time. redirects is 11,894,129 edges of this name means that page, 286,844 of which point at pages that do not exist; disambig-nodes and disambig-links are 310,529 ambiguous titles with their 723,486 candidate targets — the machinery of which Mercury you meant, laid out flat. None of it needs a server: a laptop and DuckDB will answer "which redirects dangle?" straight off the public mirror, in seconds.

Engineering

For AI engineers

who need per-record Markdown + YAML frontmatter — the hf-datasets/RAG contract — straight out of a dump, without post-cleaning wikitext residue. One call, parse_article, turns wikitext into an Article whose frontmatter already carries the link counts, categories, and canonical citation UIDs, and whose body is Markdown you can hand to a splitter without ceremony; the document profile keeps links and sections in place when you want a document rather than a training record. At corpus scale the same contract becomes selection signals: articles lets you weight or filter a training set by how well-referenced a page is before you tokenize a single byte, and every citation resolves to a CSL-JSON record when your model needs to show its sources.

For AI users

Who would like the model they talk to to have read this Wikipedia — the one where every article still knows what it cited. Most of what a language model knows about the world it learned from some rendering of the encyclopedia with the references stripped out; here the references are the point. Ask where a claim came from and the answer can be an actual record — author, title, publisher, year, a DOI where there was one — rather than a plausible-sounding invention, and the tables below are the receipts you, or your agent, can check it against. (If you are an agent reading this on someone's behalf: the YAML frontmatter was written for you. Read it before the prose.)

For Python engineers

Who want to see what regular expressions look like when someone refuses to write a parser. The citation engine is regex all the way down: hundreds of patterns composed from named building blocks with a small DSL — RegexStore, from my-basis — so one pattern can name another and have it spliced in as a subroutine call (write (?P=_x), get (?P>_x)), get wrapped in word-boundary atoms, and still compile to a single optimized program under the third-party regex engine: possessive quantifiers, atomic groups, Unicode properties, POSIX classes, and a hard 10-second deadline on every call so runaway backtracking can never hang a worker. wikiparse/data/regex.py declares seven stores — template routing, citation cleanup, identifier recognition (doi, isbn, arxiv, pmid, bibcode, and thirty-one more), latent-field extraction, contributor roles, table parsing, article translation — and the YAML files beside it hold the vocabularies they are built from. If you have ever wondered how far regex can be pushed before it breaks, this is roughly where.

For data engineers

Who have to move a full multistream dump through a few machines without losing a stream. MachineApp leases multistream offsets to workers on any number of hosts, keeps recovery state in a parent-owned SQLite manifest per host, and drains finished shards into one compact store where each stream is a checksummed zstd row — exact replay is harmless, conflicting replay is refused, and a free-space floor pauses new claims before a disk fills. The same store accepts transports from the optional Modal path, whose budget gate validates the whole dispatch before a single container starts. The public release is what came out the other end: ten Parquet tables with pinned Arrow schemas and a SHA-256 inventory, small enough to read_parquet straight off the mirror.

Science

For NLP scholars

Who need a citation-parsing testbed where the ground is at least honest. refs and templates are the census — every paired <ref> in namespace 0, whether or not it held a recognizable {{cite}}, and each of the 37 template families by parser outcome — and articles records, per page, how many citations parsed cleanly, were flagged suspect, or failed outright (0.16 % failed; 49.7 % suspect, which is a citation-style signal, not an error verdict). That is a leakage-free frame for a parser or extraction benchmark: you know exactly which spans were in scope and what the deterministic baseline made of them. What it is not is a labeled dataset — suspect is an operational label, and no accuracy is claimed here that an independent evaluation has not earned.

For philosophers of science

Who study how a claim comes to count as knowledge, and would like a corpus where the evidential chain is still attached. Wikipedia is a live experiment in collective epistemology — verifiability rather than truth, argued out in public — and wikiparse keeps the receipts: 45,003,128 observed citations, deduplicated to canonical records so the same source can be followed across every article that leans on it, plus the category system the editors maintain themselves (529,402 categories; polyhierarchical, and gloriously inconsistent — these are Wikipedia category relations, not disciplinary consensus). Which sources get reused across fields, where the references thin out, how a discipline's self-organization differs from a library's: the tables let you ask, and the dataset card tells you what they cannot answer.

For library & information scientists

Who know that a citation is a claim about a document, and would like to see 45 million of them handled with some care. Every reference is normalized to a canonical UID — author, year, and first title word when the record has an author; publisher, year, and title when it does not (about 96 % of author-less citations are recovered this way instead of dropped) — and exported as CSL-JSON, so a Wikipedia bibliography drops straight into Zotero or pandoc --citeproc. publishers is a 20,000-row projection of what Wikipedia actually sources from, clusters groups 1,511,504 exact-signature duplicates for deterministic repair, and identifier coverage (DOI, ISBN, ISSN, PMID, PMC, arXiv, bibcode, handle, OCLC, JSTOR, …) is a first-class field rather than an afterthought. This is how Wikipedia sources works, not a scholarly citation index: encyclopedic uptake, never impact.

For bibliometricians & scientometricians

Who would like to know which scholarly sources an encyclopedia leans on, without pretending that is the same thing as influence. refs carries raw and normalized author, title, publisher, and identifier fields with a parse_status on every row; publishers ranks source strings by citation-spine multiplicity; categories gives per-category citation counts and parser-flag rates, so which journals show up in which corners of Wikipedia is a join, not a project. Call it encyclopedic uptake: the data measures how Wikipedia sources works, and cannot support claims about citations between research works, author impact, or peer-review status — the publisher strings are not entity-resolved, and nobody here is ranked.

For digital humanists

Who want cultural analytics over the encyclopedia's own structure rather than over a scrape of it. The category system (529,402 categories with article and citation aggregates), the naming layer (11.9 million redirects, 310,529 disambiguation titles), and the sourcing layer (45 million citations with publisher, medium, and year fields where they parsed) are each one Parquet file, dump-pinned to enwiki-20260601 and byte-identical on two mirrors, so an analysis written today reproduces in a decade. Source cartography by publisher and decade, reception paths through redirects, the shape of a subject's category neighborhood — all of it is on the table, with the standing caveat that adjacency in a volunteer-maintained category graph is not evidence of influence, audience, or position.

For network scientists

Who see three graphs where everyone else sees ten tables. There is a category structure (categories), an article ↔ bibliographic-record bipartite graph (articles joined to refs through canonical UIDs), and a name-resolution spine (redirects plus disambig-nodes / disambig-links, with target-existence flags so dangling edges stay visible instead of vanishing) — each with its unknowns kept as rows rather than dropped. Name them carefully: this is article–bibliographic-record citation use, not the scientific citation network, and no resolved article-hyperlink graph is promised here.

For scientists in any particular field

Who want their discipline's slice of the encyclopedia and its references, with the selection written down. There is no "science corpus" in this release — on purpose: a category label is not a disciplinary verdict, and a domain pack (life science, medicine, climate, astronomy, chemistry, history & philosophy of science are the six specified) is only worth shipping once its seed categories, closure depth, exclusions, and audit sample are published beside it. None of those packs is public yet; what is public is enough to see how your field's categories are referenced, which identifier families (DOI, PMID, bibcode, …) its citations actually carry, and where the parser struggled — the honest starting point for a slice you would trust.

The enwiki-20260601 dataset

The parser is half the project. The other half is the corpus it has already produced: the complete English Wikipedia dump of 2026-06-01 (8,531,070 pages), processed end-to-end into an immutable, checksummed SQLite store, plus the cleaned descendants derived from it:

• Canonicalized citations — every {{cite ...}} / <ref> parsed into deduplicated, CSL-JSON-exportable reference records with stable UIDs and field-level structure (authors, title, publisher, date, identifiers).
• Category graph and spines — the cross-linked category tree, plus thin category/disambiguation/redirect spines for cheap filtering and navigation.
• Supergroup partitioning — a multi-label ontology that sorts articles into analytic cohorts (navigational, sports, geography, culture, specialist reference, …) with confidence and protection signals.
• Analytical warehouse — a reversible relational projection for whole-corpus queries, every row traceable back to the immutable store.

wikiparse 2606

The first technical release is robbd/wikiparse-2606, mirrored byte-for-byte at wikiparse-2606/2606 with release inventory e6f0c3ca0e61c1e1a8ccfd5be18027d61d37a3e8a1dbfb41ba4c570ccf0e9fbd and a public checksums.sha256: ten separately loadable Parquet tables, 20,637,822 rows, five source-bound analysis cards, and exact Arrow schemas.

The release is intentionally table-sized rather than a 23 GiB database image. It does not publish the immutable source store, model weights, a paper, or inline claim-to-citation bindings.

Install

pip install wikiparse        # or: uv add wikiparse
# or straight from the repository:
pip install git+https://gitlab.com/doering-ai/apps/wiki-parse.git

wikiparse renders HTML tables and lists through Pandoc, so it needs the pandoc system binary on your PATH:

# Debian/Ubuntu: apt install pandoc  |  macOS: brew install pandoc  |  Fedora: dnf install pandoc
pandoc --version

HTML-to-Markdown conversion prefers a single long-lived pandoc lua process over forking a fresh pandoc process per table/list (wikiparse.data.pandoc_transport); this needs the same binary's Lua scripting engine (pandoc --version reporting +lua, standard in official builds). It transparently falls back to the original per-call pypandoc path when that's unavailable, so this only affects throughput, not correctness.

Requires Python ≥ 3.13.

Quickstart — wikitext to Markdown in 10 lines

No CLI, no dump, no async ceremony: parse_article runs the whole pipeline on one article's wikitext and hands you a processed Article.

import wikiparse

wikitext = """
'''Coffee''' is a [[drink]] prepared from roasted [[coffee bean]]s.<ref>
{{cite web |url=https://example.org/c |title=All About Coffee}}</ref>

== History ==
Legend attributes its discovery to an Ethiopian goatherd.
"""

article = wikiparse.parse_article(wikitext, title="Coffee")
print(article.serialize())  # YAML frontmatter + corpus-oriented Markdown body

The default corpus profile is the historical full-dump contract. Its serialized document opens with a YAML frontmatter block (aliases, internal/external link counts, categories, citations, images, …) followed by the extraction-oriented body — the shape dataset tooling can read without parsing prose. That default remains byte-stable for existing dump consumers.

For document translation, opt into the preservation profile before parsing:

from wikiparse import ArticleProfile, parse_article

article = parse_article(
    wikitext,
    title="Coffee",
    profile=ArticleProfile.DOCUMENT,
)
markdown = article.serialize()

The document profile keeps ordinary internal and external links at their body positions, retains visible sections such as See also, and omits the generated Basis index/tag prefix from serialized headings. Internal links use Markdown's [alias](destination) transport with the raw MediaWiki destination and fragment; destinations containing whitespace or parentheses use the equivalent angle-delimited form [alias](<raw destination>). External links keep their absolute scheme, path, query, and fragment. The frontmatter link counters remain normalized and section-free in both profiles, so document position and corpus aggregation are available together.

The bounded document surface directly preserves emphasis, nested lists, simple tables, inline and block code, and inline and block math produced by myform's wikitext writer. Citation records are still extracted as standalone article-level records: inline <ref> positions are not reconstructed. Arbitrary templates are not expanded; recognized templates follow the existing parser routes, while unsupported templates remain recorded in failed_citations or the unhandled-template census.

Hand off to Pandoc

A serialized article is ordinary Markdown with a YAML metadata block, so Pandoc reads it directly — the body becomes a proper document tree and every frontmatter key lands in Pandoc metadata:

pandoc -f markdown -t docx  article.md -o article.docx
pandoc -f markdown -t latex article.md

That one hop reaches Pandoc's ~50 output formats. It needs no extra wikiparse flag.

Bibliographies

An article's citations frontmatter carries canonical UIDs rather than full records, so export the citation store as CSL-JSON to give --citeproc something to render:

from pathlib import Path

from wikiparse import csl

Path("refs.json").write_text(csl.dumps(article.citation_cache.values()))

printf 'nocite: |\n  @*\n' > nocite.yaml
pandoc --citeproc --bibliography=refs.json --metadata-file=nocite.yaml \
       -f markdown -t plain article.md

Each reference is keyed by the same canonical UID wikiparse uses to deduplicate citations, so [@smith-jane__1999__beans] resolves against the exported bibliography.

Why this matters. Pandoc's own mediawiki reader does not expand templates, so {{cite ...}} inside a <ref> yields an empty footnote — silently, with a success exit code (jgm/pandoc#4404). Running the wikitext through wikiparse first keeps the bibliographic record.

Design philosophy: citations are standalone, not inline-bound. wikiparse extracts every {{cite ...}} / <ref> as a standalone deduplicated reference record, not as an inline marker bound to a specific claim in the body text. This is a deliberate design choice: inline ref markers (binding claims to specific citations inline) are permanently deferred. The standalone approach achieves a significant amount of text shortening — the inline <ref> tags and their surrounding citation markup are collapsed into a compact YAML frontmatter list of canonical UIDs — and the deduplicated records are independently queryable and exportable as CSL-JSON. Citations resolve at article granularity, which is sufficient for the corpus-builder audience this tool serves.

Author-less citations are not dropped. Citations that cannot form an author-based canonical UID (notably {{cite web}}) get a publisher/site + year + title fallback UID that recovers ~96% of them.

Process a full dump

Point the wikiparse console script at a raw Wikipedia XML dump:

wikiparse --path ~/data/wikipedia/enwiki.xml

Output lands under $MY_DATA/wikipedia/ (or ~/.local/share/wikiparse/wikipedia/ when $MY_DATA is unset): articles/ holds the per-article Markdown, with categories/ and citations/ for the cross-linked category graph and canonicalized references. The run is resumable — it skips already-processed chunks and existing article files, so an interrupted run picks up where it left off.

Process a dump on persistent machines

MachineApp processes multistream dumps on one or more trusted persistent hosts while keeping recovery state in a parent-owned SQLite manifest on each host. Spawned workers receive leased streams, publish immutable result markers, and never mutate a peer's manifest. A salt-side drainer can continuously move committed shards from every host into one compact store, while a configurable free-space floor pauses new claims before a worker fills.

Start or resume processing on the worker host:

task machine:run -- \
  --dump /path/to/enwiki-pages-articles-multistream.xml.bz2 \
  --index /path/to/enwiki-pages-articles-multistream-index.txt.bz2 \
  --output /path/to/run \
  --workers 4 --max-attempts 3 --min-free-gib 5

task machine:status -- --output /path/to/run

For multiple workers, give every host the exact same dump, index, partition count, and starting manifest state, then assign each a unique zero-based index:

# host-a
task machine:run -- ... --partition-index 0 --partition-count 2

# host-b
task machine:run -- ... --partition-index 1 --partition-count 2

The persisted topology refuses an accidental resume with different ownership. A faster host can own multiple indexes through separate manifests and supervised runner processes; never point two runners at one manifest. When adding a host to an existing run, stop and recover the original runner first, checkpoint its manifest, and clone that clean manifest before assigning the new partitions; a blank manifest would forget already-completed global stream ids.

Drain every source through one process and materialize only after the aggregate ledger proves all partitions complete:

task machine:drain -- \
  --source-ready host-a:/path/to/run/spool/ready \
  --source-status host-a:/path/to/run/status.json \
  --source-ready host-b:/path/to/run/spool/ready \
  --source-status host-b:/path/to/run/status.json \
  --destination /path/to/run/shards \
  --status-destination /path/to/run/status.json \
  --store /durable/path/corpus.sqlite3 \
  --batch-size 64 \
  --min-free-gib 100

task machine:store:status -- --store /durable/path/corpus.sqlite3
task machine:store:verify -- --store /durable/path/corpus.sqlite3 --deep

# Optional compatibility export; requires space for the multi-terabyte small-file tree.
task machine:store:materialize -- \
  --store /durable/path/corpus.sqlite3 \
  --destination /large/path/final \
  --min-free-gib 100

# The loose-shard merge remains available for older runs.
task machine:merge -- --run-root /path/to/run

The runner verifies dump and index identity on resume, retries isolated stream failures, persists content-safe page errors, recovers abandoned leases, and refuses incomplete or duplicate merges.

With --store, loose shards are only bounded transfer staging. Each pass copies at most one batch per source without deleting upstream files, verifies the worker-recorded payload byte count, commits and checkpoints the compact batch, and only then sends an exact source acknowledgement. Interrupted copies, commits, checkpoints, and acknowledgements are safe to replay; --batch-size bounds transfer, memory, transaction, and cleanup pressure together. Each successful stream becomes one deterministic zstd archive row in a synchronous=FULL SQLite database; exact replay is harmless, conflicting replay is rejected, and staging is removed only after a WAL checkpoint. The compact database is the canonical full-run artifact. store materialize is an explicit compatibility mechanism that refuses incomplete stores and insufficient destination space before reconstructing the former Markdown/YAML tree in bounded batches.

Process a dump on Modal

The optional Modal path fans multistream offsets out to isolated CPU workers, checkpoints every recoverable batch in a Volume manifest, and publishes one compact, checksummed transport per successful stream. Those transports ingest directly into the same compact store used by persistent machines, without materializing or transferring the small-file output tree. Install the cloud dependency group, generate the image's locked runtime requirements, and verify the deployed environment before dispatching paid work:

uv sync --group cloud
task modal:lock
task modal:preflight

Upload the dump, run with an explicit cost ceiling, and inspect the manifest directly:

LOCAL_PATH=~/data/enwiki-pages-articles-multistream.xml.bz2 task modal:upload

# For public dumps, stage inside Modal instead of spending local uplink bandwidth.
task modal:stage-url -- \
  --source-url https://dumps.wikimedia.org/enwiki/YYYYMMDD/enwiki-YYYYMMDD-pages-articles-multistream.xml.bz2 \
  --name enwiki-YYYYMMDD-pages-articles-multistream.xml.bz2 \
  --expected-bytes 123456789 \
  --expected-sha256 0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef \
  --parts 16
# Start with one bounded stream and inspect its observed duration/cost.
DUMP_NAME=enwiki-pages-articles-multistream.xml.bz2 \
  task modal:run -- \
    --index-path ~/data/enwiki-pages-articles-multistream-index.txt.bz2 \
    --max-streams 1 --budget-usd 0.05

# Full dispatch requires an operator-approved ceiling; completed smoke work is skipped.
DUMP_NAME=enwiki-pages-articles-multistream.xml.bz2 \
  task modal:run -- \
    --index-path ~/data/enwiki-pages-articles-multistream-index.txt.bz2 \
    --budget-usd "$APPROVED_BUDGET_USD"
task modal:status

--max-streams N bounds a smoke run; --index-path PATH can use a local index without uploading it. The budget gate validates the entire selected dispatch before containers start, and successful offsets are durable checkpoints, so repeating the command processes only unfinished or failed streams. The estimate is deliberately conservative and is not a substitute for the provider's billing dashboard.

To add Modal as a burst alongside one unpartitioned machine runner, stop and recover the runner before dispatch and use the same full stream map, source identity, and parser build in both authorities. Partitioned authority composition is deliberately refused until one aggregate task map can be proven across every owner:

export WIKIPARSE_MODAL_VOLUME=wikiparse-enwiki-YYYYMMDD-capture-v2
DUMP_NAME=enwiki-YYYYMMDD-pages-articles-multistream.xml.bz2 \
  task modal:run -- \
    --index-path /path/to/enwiki-YYYYMMDD-pages-articles-multistream-index.txt.bz2 \
    --machine-manifest-path /path/to/stopped/manifest.sqlite3 \
    --max-streams 5000 --budget-usd 20

modal volume get "$WIKIPARSE_MODAL_VOLUME" transports /path/to/download
task machine:store:adopt-campaign-transports -- \
  --store /durable/path/corpus.sqlite3 \
  --transports /path/to/download/transports \
  --campaign /path/to/campaign.sqlite3 \
  --machine-manifest-path /path/to/stopped/manifest.sqlite3 \
  --keep

Composed transport adoption commits 64 campaign-validated archives per transaction by default, records campaign authority as per-stream ingress provenance, and only then marks the stopped machine manifest done. The canonical store remains machine-owned, binds the complete stream map, and can therefore accept later local loose shards without weakening paid-ingress checks. Only successful, packet-complete, checksum-verified transports are adopted, and their transport files are retained as replay evidence. The runner refuses foreign source/build/coordinates, partitioned or actively leased manifests, missing campaign receipts, and conflicting replays. Restart the reserved runner only after the adopted manifest and its atomic status snapshot are back in place.

The implementation follows Modal's current guidance for batched .map() processing, Volume reload and commit semantics, and source-mounted images.

Documentation

• API reference & guides: the docs/ tree (built with Sphinx + MyST).
• Architecture: five subpackages — data (shared regex/resource plumbing), types (wikitext primitives), citations (reference parsing/canonicalization), pages (article/category translation), cli (the pipeline driver).

See CHANGELOG.md for release history.

Appendix: screenshots

Every figure above, in one place, plus the live surfaces they were taken from. Capture notes for each file live in assets/screenshots/README.md.

Live surfaces:

• robbd/wikiparse-2606 on Hugging Face — the dataset viewer for all ten configs.
• wikiparse-2606/2606 on DigitalOcean Spaces — the byte-identical mirror, with checksums.sha256 and the five analysis cards (category structure, citation quality, provenance repair, science closure, wave-0 landscape).
• wikiparse/data/regex.py — the regex stores, in source.