mwextract

Convert a MediaWiki XML dump (as shipped by Fandom/Wikia and other MediaWiki sites) into clean, structured Markdown, JSON, or XML — one command, no pandoc, no other binaries.

It uses only the Wikimedia parsing stack:

• mwxml streams pages out of the dump (memory-safe, namespace-aware).
• mwparserfromhell parses wikitext: infobox/template fields become metadata, and the article body is rendered to clean Markdown.
• py7zr extracts the .7z archive the dump usually comes in.

Install

uv add mwextract
# or
pip install mwextract

Get a dump

mwextract does not download anything — you fetch the dump yourself. Fandom wikis expose current-page dumps under a URL like:

https://s3.amazonaws.com/wikia_xml_dumps/<x>/<xx>/<wiki>_pages_current.xml.7z

For example, the Game of Thrones wiki: https://s3.amazonaws.com/wikia_xml_dumps/g/ga/gameofthrones_pages_current.xml.7z

Download the .7z and point mwextract at it.

Use it from Python

import mwextract

result = mwextract.convert(
    "gameofthrones_pages_current.xml.7z",   # .7z, .xml, or .xml.bz2
    "out/",                                  # output directory
    base_url="https://gameofthrones.fandom.com/wiki/",  # optional
    fmt="md",                                # "md" | "json" | "xml"
    layout="combined",                       # "combined" | "per-file"
)

print(result.pages, "pages ->", result.outputs)

convert() extracts the archive to a temporary directory (removed afterwards), streams every main-namespace, non-redirect article, and writes the chosen format.

Use it from the command line

mwextract gameofthrones_pages_current.xml.7z out/ \
  --base-url https://gameofthrones.fandom.com/wiki/ \
  --format md \
  --layout combined

usage: mwextract [-h] [-b BASE_URL] [-f {md,json,xml}]
                 [-l {combined,per-file}] [-q] [-V]
                 dump output_dir

Output

Layouts

• combined (default) — a single file for the whole corpus: corpus.md, corpus.json (a JSON array), or corpus.xml (<pages><page>…</page></pages>).
• per-file — one file per article, named after the title: Daenerys_Targaryen.md, .json, or .xml.

Both layouts are written incrementally as pages stream in, so memory stays flat no matter how large the dump is.

Formats

Markdown — YAML front-matter (title, url, infobox fields) + an optional summary line + the rendered body:

---
title: "Jon Snow"
url: https://gameofthrones.fandom.com/wiki/Jon_Snow
house: "Stark"
allegiance: "Night's Watch"
---

# Jon Snow

> **House:** Stark · **Allegiance:** Night's Watch

Jon Snow is the illegitimate son of ...

JSON — the same content, structured:

{
  "title": "Jon Snow",
  "url": "https://gameofthrones.fandom.com/wiki/Jon_Snow",
  "metadata": { "house": "Stark", "allegiance": "Night's Watch" },
  "body": "Jon Snow is the illegitimate son of ..."
}

XML — the same content as elements:

<page>
  <title>Jon Snow</title>
  <url>https://gameofthrones.fandom.com/wiki/Jon_Snow</url>
  <metadata>
    <field name="house">Stark</field>
    <field name="allegiance">Night's Watch</field>
  </metadata>
  <body>Jon Snow is the illegitimate son of ...</body>
</page>

body is the rendered Markdown body only — the structured metadata/url carry everything the Markdown front-matter would, so nothing is duplicated.

What gets kept / dropped

• Kept: main-namespace articles, section hierarchy, bold/italic, list items, wikilink and external-link visible text, infobox fields (as metadata).
• Dropped: redirects and non-article namespaces, file/image/category/media links, citation footnotes (<ref>), templates (after their infobox fields are extracted), and comments.

License

MIT © Aditya Parab