wmlinksfromhell

wmlinksfromhell is a Python package for working with links on Wikimedia and other MediaWiki sites. It is mainly focused on Wikimedia interwiki links, language and project prefixes, and Wikimedia URLs, with support for local links and external URLs too.

It can resolve links to their destination wiki and page, convert them between interwiki and URL forms, and give you the details you need to work with them in your code.

Developed by Max.

Full documentation is available on Read the Docs. Development takes place on GitHub, and bugs and feature requests can be reported in the issue tracker.

Project links

• Documentation
• API Reference
• Changelog
• Issues
• Repository

Install

python3 -m pip install wmlinksfromhell

Pywikibot support is optional:

python3 -m pip install 'wmlinksfromhell[pywikibot]'

Supports Python 3.9 and newer.

Quick start

Get information about a wiki:

import wmlinksfromhell

wiki = wmlinksfromhell.wiki("enwiki")

print(wiki.language)       # en
print(wiki.hostname)       # en.wikipedia.org
print(wiki.page_url("Apple"))
print(wiki.interwiki("Apple"))

Resolve a Wikimedia link:

result = wmlinksfromhell.resolve("w:en:Apple")

print(result.dbname)        # enwiki
print(result.title)         # Apple
print(result.url)           # https://en.wikipedia.org/wiki/Apple

Use the full project name for project/language interwiki output when needed:

result = wmlinksfromhell.resolve_url(
    "https://hi.wikipedia.org/wiki/Apple",
    long_project_prefix=True,
)

print(result.canonical_interwiki)  # wikipedia:hi:Apple

The short form remains the default, and wikipedia:hi:Apple is accepted as input.

Parse links from MediaWiki text:

import wmlinksfromhell

code = wmlinksfromhell.parse(
    "[[w:en:Apple]] https://de.wikipedia.org/wiki/Berlin https://example.org/",
    source="metawiki",
)

for link in code.filter_links():
    print(link.original)
    print(link.dbname, link.title, link.url)

The parser keeps the original way a link was written separate from the destination it resolves to. This lets you work with the same destination whether it came from an interwiki link, a wiki URL, or another supported form.

Wikimedia links

The resolver understands Wikimedia language and project prefixes, including prefix chains:

w:en:Apple
wikt:simple:uppercase
b:de:Main Page
:de:q:Hauptseite
:m:en:About
:w:it:b:Wiskunde

It also handles Wikimedia URL forms such as article paths and index.php, including fragments, revisions, diffs, actions, and page IDs.

For example:

result = wmlinksfromhell.resolve(
    "https://en.wikipedia.org/w/index.php?title=Apple&oldid=123456",
)

print(result.dbname)       # enwiki
print(result.revision)     # 123456
print(result.interwiki)    # w:en:Special:PermanentLink/123456

A leading colon changes MediaWiki's special handling for some links, but it does not necessarily change the destination. For example, [[en:Apple]] and [[:en:Apple]] both point to English Wikipedia's Apple page; the leading-colon form is used when the link should be shown as a normal link in the page.

Local and external links

Local targets are resolved when a source wiki is available:

result = wmlinksfromhell.resolve(
    "Apple",
    source="enwiki",
)

print(result.dbname)       # enwiki
print(result.title)        # Apple

Ordinary external URLs are kept as external destinations:

result = wmlinksfromhell.resolve("https://example.org/page")

print(result.status)             # ResolutionStatus.EXTERNAL
print(result.destination_type)   # DestinationType.EXTERNAL
print(result.url)

Finding links

filter_links() can select links by the things you care about:

code.filter_links(dbname="enwiki")
code.filter_links(family="wikipedia", language="en")
code.filter_links(namespace="MediaWiki")
code.filter_links(hostname="en.wikipedia.org")

code.filter_links(wiki_only=True)
code.filter_links(interwiki_only=True)
code.filter_links(local_only=True)
code.filter_links(external_only=True)
code.filter_links(dbname_in={"enwiki", "dewiki"})

code.destinations returns the resolved destinations for parsed links.

Links inside HTML comments are ignored by default. Include them when needed:

code.filter_links(comment_links=True)

Working with a link

Links expose both the original link and its resolved destination:

for link in code.filter_links():
    print(link.original)
    print(link.dbname)
    print(link.title)
    print(link.interwiki)
    print(link.url)
    print(link.destination)

Links can also be matched and converted:

link.matches(dbname="enwiki", title="Apple")

link.set_url()
link.set_interwiki()
link.set_local(source="enwiki")
link.set_label("New label")

Metadata and serialization

Metadata can be enumerated without reaching into private state:

metadata = wmlinksfromhell.MetadataStore()
wmlinksfromhell.wikis(metadata)
metadata.wikis_for_family("wikipedia")
metadata.languages()
metadata.canonical_namespaces("enwiki")
metadata.interwiki_map("enwiki").prefixes()

Destination objects can be reconstructed from as_dict() output, and ResolutionResult objects can be serialized directly with as_json().

Command line

Resolve a value from the command line:

python -m wmlinksfromhell resolve "w:en:Apple"
python -m wmlinksfromhell resolve "Apple" --source enwiki --json

Pywikibot

When Pywikibot is installed, a Page can be passed directly to parse_page():

import pywikibot
import wmlinksfromhell

site = pywikibot.Site("en", "wikipedia")
page = pywikibot.Page(site, "Apple")

code = wmlinksfromhell.parse_page(page)

for link in code.filter_links():
    print(link.original, link.url)

Metadata

Wikimedia SiteMatrix and interwiki metadata can be loaded and cached locally. The lower-level MetadataStore, Resolver, and Destination APIs are available when you need more control.

See the API Reference for the full API.