mtgwiki 1.2.0

mtgwiki is a small Python package for getting useful, reusable data out of MediaWiki Action API sites without teaching the package what the wiki's subject matter means. MTG Wiki is the default endpoint and the original use case, but the transport, traversal, parsing and snapshot layers are domain-neutral.

The package does not contain functions such as get_plane(), get_character(), flora() or depicted_cards(). Instead it exposes generic MediaWiki building blocks and a generic snapshot/structure layer that other projects can interpret however they want.

Design rule

If a feature needs to know Magic semantics, it does not belong in mtgwiki.

The package has three jobs:

1. Acquire data politely and efficiently from MediaWiki.
2. Structure generic wiki syntax such as templates, parameters, sections, lists, tables and links.
3. Export JSON-friendly snapshots that preserve source/provenance and raw data.

Install

From the unpacked project folder:

py -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install -e .

For sustained API use, identify your own project in the User-Agent:

from mtgwiki import Wiki

wiki = Wiki(
    user_agent="my-project/0.1 (https://example.com/contact)"
)

Point the same client at another MediaWiki installation by supplying its Action API endpoint:

wiki = Wiki(
    api_url="https://www.mediawiki.org/w/api.php",
    user_agent="my-project/0.1 (https://example.com/contact)",
)

The package name/default URL are conveniences; the public data model contains no Magic-specific entity types.

Compatibility

The client targets modern MediaWiki Action API installations and always requests JSON with formatversion=2. Revision-content handling supports main-slot responses and includes fallbacks for older response shapes. Rendered section discovery prefers tocdata and falls back to the older sections output. Use siteinfo() and paraminfo() when a consuming project needs to discover site-specific capabilities rather than assume them.

Quick start

from mtgwiki import Wiki

wiki = Wiki()

page = wiki.get("Skyship Weatherlight")
print(page["title"])

hits = wiki.search("Bloomburrow", limit=20)
for hit in hits:
    print(hit["title"])

Search results are ranked search results. search() does not pretend the first result is the entity you meant.

Generic structure discovery

Ask the package what syntax exists instead of hard-coding what you expect:

structure = wiki.structure(
    "Skyship Weatherlight",
    section="In-game references",
)

for template in structure["templates"]:
    print(template["name"])
    print(template["params"].keys())

A template call is returned generically, for example:

{
    "name": "Some template",
    "depth": 1,
    "raw": "{{Some template|x=1|data=...}}",
    "params": {
        "x": "1",
        "data": "...",
    },
    "params_clean": {
        "x": "1",
        "data": "...",
    },
    "parameters": [...],
}

The package does not assign meaning to x, data, art, species, job1 or any other parameter name.

Snapshots: easiest way to build reusable datasets

snapshot = wiki.extract("Dack Fayden")

print(snapshot.keys())
print(snapshot["source"])
print(snapshot["data"])
print(snapshot["structure"].keys())

A snapshot contains:

schema_version
exists
source
  api_url
  requested_title
  resolved_title
  pageid
  revision_id
  revision_timestamp
  retrieved_at
  redirects
page
revision
data
  categories
  links
  templates
  images
  external_links
  pageprops
content
  raw
  text
structure
  sections
  templates
  wikilinks
  external_links
  lists
  tables
raw
  page
  normalized
  redirects
  converted

Raw source is deliberately retained. Cleaned/normalized values are conveniences, not replacements for the source.

Batch extraction

MediaWiki accepts multiple titles in one query. extract_many() batches up to 50 titles per request group instead of making one independent request per title.

pages = wiki.extract_many(
    [
        "Urza",
        "Dack Fayden",
        "Skyship Weatherlight",
        "Bloomburrow (plane)",
    ]
)

You can request a smaller snapshot when you do not need everything:

pages = wiki.extract_many(
    ["Urza", "Dack Fayden"],
    include=("revision", "categories", "wikitext", "structure"),
)

Supported include values:

revision
categories
links
templates
images
external_links
pageprops
wikitext
structure

JSON / JSONL

from mtgwiki import write_json, write_jsonl

write_json("one-page.json", wiki.extract("Urza"))
write_jsonl("dataset.jsonl", wiki.extract_many(["Urza", "Karn", "Squee"]))

JSONL is convenient for larger datasets because every page is one independent line.

Raw API is always available

The high-level helpers never lock you out of MediaWiki itself:

data = wiki.api(
    action="query",
    meta="siteinfo",
    siprop="general|namespaces",
)

Continuation:

for response in wiki.iter_api(
    action="query",
    list="categorymembers",
    cmtitle="Category:Example",
    cmlimit="max",
):
    print(response)

Or collect common list modules directly:

members = wiki.members("Example category")
backlinks = wiki.backlinks("Example page")
embeds = wiki.embedded_in("Template:Example")

These are MediaWiki concepts, not domain semantics.

Lazy list iteration

For large list modules you can stream items instead of building one large list in memory:

for item in wiki.iter_list(
    "allpages",
    aplimit="max",
    apnamespace=0,
):
    print(item["title"])

list() remains available when collecting everything is more convenient.

Recursive categories

Category trees are a generic MediaWiki structure. members() can now traverse them without every research script reimplementing a queue and cycle guard:

pages = wiki.members(
    "Example root category",
    recurse=True,
    namespace=0,
    cmtype="page",
)

for page in pages:
    print(page["title"])

Use an integer to limit depth. recurse=1 includes members of immediate subcategories but does not descend further:

pages = wiki.members(
    "Example root category",
    recurse=1,
    namespace=0,
    cmtype="page",
)

For very large category trees, use iter_members() to stream the same traversal lazily. Recursive traversal is cycle-safe and de-duplicates members that appear through multiple category paths.

Sections

For rendered section discovery the package asks MediaWiki for tocdata and falls back to the older sections response when necessary:

for section in wiki.sections("Bloomburrow (plane)"):
    print(section["line"])

Fetch only one named section:

text = wiki.section("Bloomburrow (plane)", "Flora")

Extract generic list items without losing the original markup:

items = wiki.section_items(
    "Bloomburrow (plane)",
    "Flora",
    clean=True,
)

for item in items:
    print(item["raw_text"])
    print(item["text"])

The method knows only that it is reading a list in a section. It does not know what "Flora" means.

Wikitext structure parser

The built-in parser is deliberately conservative and dependency-free. It discovers:

• section headings
• nested template calls and arbitrary parameters
• wikilinks
• external links
• list items and nesting depth
• MediaWiki tables

Use it on any string:

from mtgwiki import parse_wikitext

structure = parse_wikitext("{{Thing|a=1}}\n* [[Target|Label]]")

It is not intended to reproduce MediaWiki's renderer. When exact rendered output matters, use wiki.parse() / wiki.section() and let the server parse the page. Raw source and spans are kept so a later project can always fall back to the original material.

Revisions and provenance

extract() records the revision ID, timestamp and SHA-1 where available. This makes exported data traceable to the wiki revision it came from.

snapshot = wiki.extract("Urza", include=("revision", "wikitext"))
print(snapshot["source"]["revision_id"])
print(snapshot["revision"]["sha1"])

API capability discovery

Do not assume every MediaWiki installation exposes exactly the same features:

info = wiki.siteinfo()
params = wiki.paraminfo(["query", "parse"])

Polite API behavior

The client defaults to:

serial requests only
min_interval = 0.5 seconds
maxlag = 5
Retry-After support
exponential retry/backoff
memory cache

Statistics:

print(wiki.client.stats())

Optional persistent cache:

wiki = Wiki(
    cache_path=".mtgwiki-cache.sqlite",
    cache_ttl=24 * 60 * 60,
)

The SQLite cache is optional and uses only Python's standard library.

Compatibility helpers

The 1.0 helpers infobox(), infobox_fields(), templates_with_params() and template_params() remain available so existing experiments keep working. New code should generally prefer template_calls() or structure() because those do not assume a particular template convention.

Tests

Offline unit tests:

python -m unittest discover -s tests -v

Then run the supplied live verifier:

python verify_live.py

It creates:

mtgwiki_verify_report.json
verify_output/pages.jsonl

Send mtgwiki_verify_report.json back to ChatGPT if you want the live result reviewed.