Skip to main content

Wikipedia tools (for Humans)

Project description

https://img.shields.io/pypi/v/wptools.svg https://img.shields.io/badge/license-MIT-green.svg

Python and command-line MediaWiki access for Humans.

  • get an HTML or plain text “extract” (lead or summary)

  • get a representative image (pageimage, thumb, etc.)

  • get an Infobox as a python dictionary

  • get any/all Wikidata by title

  • get info in any language

  • get random info

This package is intended to make it as easy as possible to get data from MediaWiki instances, expose more Wikidata, and extend the MediaWiki API just for kicks. We say “(for Humans)” because that is a goal. Please see NOTES for details. Questions, feedback, and especially contributions are welcome!

Install

$ pip install wptools
✨🦄✨

Usage

>>> import wptools

An instance can be initialized by:

wptools.page (self)

  • None: <NoneType>

  • lang: <str> MediaWiki language code (default=’en’)

  • pageid: <int> a MediaWiki Page ID

  • title: <unicode> a MediaWiki article title

  • wiki: <str> any MediaWiki site

  • wikibase: <str> Wikidata item entity ID

The simplest way to begin is with a title:

>>> f = wptools.page("Flannery O'Connor")
Flannery_O'Connor (en)
{
  lang: en
  title: Flannery_O'Connor
}

The default language is ‘en’ (English):

>>> t = wptools.page('穐吉敏子')
穐吉敏子 (en)
{
  lang: en
  title: 穐吉敏子
}

Leaving off arguments invokes a random lookup in English:

>>> r = wptools.page()
en.wikipedia.org (random) None
Sylvia_Rivera (en)
{
  lang: en
  pageid: 3296309
  title: Sylvia_Rivera
}

If you give only lang, you get a random article in that language:

>>> zh = wptools.page(lang='zh')
zh.wikipedia.org (random) None
哈莉特·塔布曼 (zh)
{
  lang: zh
  pageid: 211070
  title: 哈莉特·塔布曼
}

You can also start with a wikibase item:

>>> q = wptools.page(wikibase='Q43303')
Q43303 (en)
{
  lang: en
  wikibase: Q43303
}

Or, another wiki site:

>>> m = wptools.page(wiki='en.wikiquote.org')
en.wikiquote.org (random) None
Malala_Yousafzai (en)
{
  lang: en
  pageid: 146817
  title: Malala_Yousafzai
  wiki: en.wikiquote.org
}

Instance attributes echo automatically. You can turn that off with silent=True:

>>> r = wptools.page(silent=True)

Request details echo to stderr with verbose=True:

>>> r = wptools.page(verbose=True)

You can set a timeout (in seconds) on any instance:

>>> r.set_timeout(5)

You can set a proxy server like this:

>>> wptools.set_proxy('http://example.com:80')

Examples

Get a representative image:

>>> frida = wptools.page('Frida Kahlo').get_query()
en.wikipedia.org (query) Frida_Kahlo
en.wikipedia.org (imageinfo) File:Frida Kahlo, by Guillermo Kahlo.jpg|Fi...

>>> frida.image('page')['url']
u'https://upload.wikimedia.org/wikipedia/commons/0/06/Frida_Kahlo%2C_by_Guillermo_Kahlo.jpg'

>>> frida.image('thumb')['url']
u'https://upload.wikimedia.org/wikipedia/commons/thumb/0/06/Frida_Kahlo%2C_by_Guillermo_Kahlo.jpg/160px-Frida_Kahlo%2C_by_Guillermo_Kahlo.jpg'
https://upload.wikimedia.org/wikipedia/commons/thumb/0/06/Frida_Kahlo%2C_by_Guillermo_Kahlo.jpg/160px-Frida_Kahlo%2C_by_Guillermo_Kahlo.jpg

Note: A page’s image can come from the pageimage or thumbnail (via get_query()), from an Infobox (via get_parse()), from Wikidata Property:P18 (via get_wikidata()), or from the RESTBase image or thumb (via get_rest()). All images are stored in the images attribute, and can be queried with the image() method.

Get a text (or HTML) extract:

>>> ella = wptools.page('Ella Fitzgerald').get_query()
en.wikipedia.org (query) Ella_Fitzgerald

>>> print ella.extext
**Ella Jane Fitzgerald** (April 25, 1917  June 15, 1996) was an
American jazz singer often referred to as the First Lady of Song,
Queen of Jazz and Lady Ella. She was noted for her purity of tone,
impeccable diction, phrasing and intonation, and a "horn-like"
improvisational ability, particularly in her scat singing.
...

>>> print ella.extract
<p><b>Ella Jane Fitzgerald</b> (April 25, 1917  June 15, 1996) was an
American jazz singer often referred to as the First Lady of Song,
Queen of Jazz and Lady Ella. She was noted for her purity of tone,
impeccable diction, phrasing and intonation, and a "horn-like"
improvisational ability, particularly in her scat singing.</p>
...

Get an Infobox as a python object:

>>> fela = wptools.page('Fela Kuti').get_parse()
en.wikipedia.org (parse) Fela_Kuti

>>> fela.infobox['instrument']
'Saxophone, vocals, keyboards, trumpet, guitar, drums'

Note: Getting data from Infoboxen may be unavoidable, but getting Wikidata (via get_wikidata()) is preferred. Wikidata is structured but (sometimes) data poor, while Infoboxen are unstructured and (frequently) data rich. Please consider updating Wikidata if the information you want is only available in a MediaWiki instance so that others may benefit from open, linked data.

Get an (album, book, film, etc.) cover image:

>>> blue = wptools.page('Blue Train (album)').get_parse()

>>> blue.image('cover')['url']
u'https://upload.wikimedia.org/wikipedia/en/6/68/John_Coltrane_-_Blue_Train.jpg'
https://upload.wikimedia.org/wikipedia/en/6/68/John_Coltrane_-_Blue_Train.jpg

Get wikidata by title:

>>> fry = wptools.page('Stephen Fry').get_wikidata()
www.wikidata.org (wikidata) Stephen_Fry
www.wikidata.org (wikidata) Q8817795|Q5|Q7066|Q145
Stephen_Fry (en)
{
  cache: <dict(2)> {claims, wikidata}
  claims: <dict(4)> {Q145, Q5, Q7066, Q8817795}
  description: English comedian, actor, writer, presenter, and activist
  images: <dict(1)> {wikidata-image}
  label: Stephen Fry
  lang: en
  modified: 2016-10-31T22:02:03Z
  props: <dict(8)> {P135, P18, P27, P31, P345, P569, P856, P910}
  title: Stephen_Fry
  wikibase: Q192912
  wikidata: <dict(8)> {IMDB, birth, category, citizenship, image, in...
  wikidata_url: https://www.wikidata.org/wiki/Q192912
}

Note: Resolved properties and claims are stored in the wikidata attribute. Wikidata claims are processed via WPTools._WIKIPROPS. Properties (e.g. P17: country) are stored in props and those properties that have Wikidata entities for values (e.g. Q142 instead of France) are stored in claims and resolved by another Wikidata API call (as shown above).

Extend Wikidata claims to be resolved:

>>> simone = wptools.page('Simone de Beauvoir')
>>> simone._WIKIPROPS['P21'] = 'gender'

>>> simone.get_wikidata()
www.wikidata.org (wikidata) Simone_de_Beauvoir
www.wikidata.org (wikidata) Q142|Q5|Q1214721|Q7066|Q8745624|Q6581072

>>> simone.wikidata['gender']
'female'

Get special lead section HTML:

>>> buddha = wptools.page('Buddha').get_rest()
en.wikipedia.org (/page/mobile-text/) Buddha

>>> buddha.lead
<img query-thumbnail src="https://upload.wikimedia.org/wikipedia/commons...
<span heading><a href="https://en.wikipedia.org/wiki/Gautama_Buddha">Gau...
<span snipped><span><b>Gautama Buddha</b>, also known as <b>Siddhārtha G...
Gautama taught a <a href="https://en.wikipedia.org/wiki/Middle_Way" titl...
Gautama is the primary figure in Buddhism. He is recognized by Buddhists...
<span metadata>Modified: 2016-10-13T09:44:13Z</span>

Note: The lead attribute contains the assembled stand-alone, encyclopedia-like HTML fragment:

  • <img {source-type}> selected image

  • <span heading> wiki-linked title and description

  • <span snipped> lead paragraphs with (noprint, reference, &c.) snipped

  • <span metadata> available metadata (e.g. Last modified)

Get all the things:

>>> jill = wptools.page('Jill Lepore').get()
en.wikipedia.org (query) Jill_Lepore
en.wikipedia.org (parse) 22469182
www.wikidata.org (wikidata) Q6192915
www.wikidata.org (wikidata) Q30|Q5
Jill_Lepore (en)
{
  cache: <dict(4)> {claims, parse, query, wikidata}
  claims: <dict(2)> {Q30, Q5}
  description: American historian
  extext: <str(1016)> **Jill Lepore** (born August 27, 1966) is an A...
  extract: <str(1114)> <p><b>Jill Lepore</b> (born August 27, 1966) ...
  infobox: <dict(38)> {academic_advisors, alma_mater, alt, author_ab...
  label: Jill Lepore
  lang: en
  modified: 2016-08-11T19:53:32Z
  pageid: 22469182
  parsetree: <str(50677)> <root><template><title>Infobox scientist</...
  props: <dict(3)> {P27, P31, P569}
  random: Ramesh Bidhuri
  title: Jill_Lepore
  url: https://en.wikipedia.org/wiki/Jill_Lepore
  urlraw: https://en.wikipedia.org/wiki/Jill_Lepore?action=raw
  wikibase: Q6192915
  wikidata: <dict(3)> {birth, citizenship, instance}
  wikidata_url: https://www.wikidata.org/wiki/Q6192915
  wikitext: <str(22540)> {{Infobox scientist| name = Jill Lepore| na...
}

Query results are cached in the cache attribute:

>>> jill.cache['query']['info']
{'bytes': 1911.0,
 'content': 'application/json; charset=utf-8',
 'kB/s': '2.6',
 'seconds': '0.743',
 'status': 200,
 'url': 'https://en.wikipedia.org/w/api.php?action=query&exintro&inprop=displaytitle|url|watchers&list=random&pithumbsize=240&ppprop=wikibase_item&prop=extracts|images|info|pageimages|pageprops&redirects&rnlimit=1&rnnamespace=0&titles=Jill_Lepore',
 'user-agent': 'wptools/0.1.7 (https://github.com/siznax/wptools) PycURL/7.43.0 libcurl/7.49.1 SecureTransport zlib/1.2.8'}

The wptools user-agent will look like this:

wptools/version (https://github.com/siznax/wptools) pycurl libs

Methods

Get help on instance methods like this:

>>> help(wptools.core)
>>> help(<instance>)

get (self)

make all requests necessary to populate all the things

  • get_query()

  • get_parse()

  • get_wikidata()

get_claims (self)

Wikidata:API action=wbgetentities for labels of claims

  • e.g. {‘Q298’: ‘country’} resolves to {‘country’: ‘Chile’}

  • use get_wikidata() to populate claims

get_imageinfo (self)

MediaWiki request for API:Imageinfo

  • images: <dict> updates image URLs, sizes, etc.

get_parse (self)

MediaWiki:API action=parse request for:

  • images: <dict> {parse-image, parse-cover}

  • infobox: <dict> Infobox data as python dictionary

  • links: <list> interwiki links (iwlinks)

  • pageid: <int> MediaWiki database ID

  • parsetree: <unicode> XML parse tree

  • wikibase: <unicode> Wikidata entity ID or wikidata URL

  • wikitext: <unicode> raw wikitext URL

get_query (self)

MediaWiki:API action=query request for:

  • extext: <unicode> plain text (Markdown) extract

  • extract: <unicode> HTML extract via Extension:TextExtract

  • images: <dict> {query-pageimage, query-thumbnail}

  • pageid: <int> MediaWiki database ID

  • random: <unicode> a random article title with every request!

  • url: <unicode> the canonical wiki URL

  • urlraw: <unicode> ostensible raw wikitext URL

get_random (self)

MediaWiki:API action=query request for:

  • pageid: <int> MediaWiki database ID

  • title: <unicode> article title

get_rest (self)

RESTBase /page/mobile-text/ request for:

  • description: <unicode> apparently, Wikidata description

  • images: <dict> {rest-image, rest-thumb}

  • lead: <str> encyclopedia-like lead section

  • modified: <str> ISO8601 date and time

  • url: <unicode> the canonical wiki URL

  • urlraw: <unicode> ostensible raw wikitext URL

get_wikidata (self)

Wikidata:API action=wbgetentities request for:

  • claims: <dict> Wikidata claims (to be resolved)

  • description: <unicode> Wikidata description

  • images: <dict> {wikidata-image} Wikidata Property:P18

  • label: <unicode> Wikidata label

  • modified: <str> ISO8601 date and time

  • props: <dict> Wikidata properties

  • wikibase: <str> Wikidata URL

  • wikidata: <dict> resolved Wikidata properties and claims

image (self, token)

Returns first image info with kind containing token

show (self)

Pretty-print instance attributes.

wptool

Basic functionality on the command-line is provided by the wptool command.

$ wptool -h
usage: wptool [-h] [-H] [-l L] [-n] [-q] [-s] [-t T] [-v] [-w W]

Get Wikipedia article info and Wikidata via MediaWiki APIs.

Gets a random English Wikipedia article by default, or in the
language -lang, or from the wikisite -wiki, or by specific
title -title. The output is a plain text extract unless -HTML.

optional arguments:
  -h, --help      show this help message and exit
  -H, -HTML       output HTML extract
  -l L, -lang L   language code
  -n, -nowrap     do not wrap text
  -q, -query      show query and exit
  -s, -shh        quiet output to stderr
  -t T, -title T  get a specific title
  -v, -verbose    HTTP status to stderr
  -w W, -wiki W   use alternative wikisite

Powered by https://github.com/siznax/wptools/

For example:

$ wptool -t "Jeanne d'Arc" -l fr -s
JEANNE_D'ARC—sainte et héroïne de l'histoire de France

![Jeanne d'Arc](https://upload.wikimedia.org/wikipedia/commons/3/39/...)

**Jeanne d'Arc**, née vers 1412 à Domrémy village du duché de Bar dont
une partie relevait du royaume de France pour le temporel et de
l'évêché de Toul pour le spirituel (actuellement dans le département
des Vosges en Lorraine), et morte sur le bûcher le 30 mai 1431 à
Rouen, capitale du duché de Normandie alors possession du royaume
d'Angleterre, est une héroïne de l'histoire de France, chef de guerre
et sainte de l'Église catholique, surnommée depuis le XVIe siècle «
_la Pucelle d'Orléans_ » et, depuis le XIXe siècle, « _mère de la
nation française_ ».
...

<https://fr.wikipedia.org/wiki/Jeanne_d%27Arc>
<https://www.wikidata.org/wiki/Q7226>

Please enjoy!

@siznax 👹

Release History

0.1.8 (2016-11-01)

  • Many fixes with help of @seansay (thank you!)

  • Consolidated images and caches

  • Automatically get_imageinfo() via MediaWiki API:Imageinfo

  • Added timeout support via set_timeout()

  • Added proxy support via set_proxy()

  • Support for non-https wikis via wiki=’http://example.com

  • Support for language variants (e.g. variant=’zh-cn’)

  • Support get_query() by pageid

  • Raise LookupError on API missing page/title

0.1.7 (2016-09-20)

  • Minor fixes

  • Provide local SSL certs with python-certifi

  • Enabled extending Wikidata properties

  • Improved cache checking

0.1.4 (2016-09-06)

  • Python 3 support!

  • Implemented get_claims()

  • Patch get_rest() path, tests/test_advanced

  • Implemented get_wikidata() by title, lang

  • Geo coordinates Property:P625 from Wikidata

  • Added RESTBase support, lead attribute

  • Made wptools.page() the atomic object

0.1.0 (2016-08-25)

  • Many fixes, enhancements.

  • ONE CLI SCRIPT TO RULE THEM ALL: wptool.

  • Much testing of random (title, lang, and wiki).

  • Allow chaining get_s or just get().

  • Enabled full-hydration from wikibase only.

  • Block further requests on fatal error.

  • Get 240x thumbnails instead of default 50x.

  • Harmonized silent and verbose arguments.

  • Mock-up tests/test_advanced.py.

  • Made NOTES.md a knowledge base.

  • Use _humans_ in README examples. ;-)

0.0.5 (2016-08-23)

  • Major re-write.

  • Exposed core.WPTools as entrypoint.

  • Added get_parse(), get_query(), and get_wikidata().

  • Added get(self) to query all APIs.

  • Added show(self) method to display fetched attrs.

  • Show instance attributes after each request.

  • Ignore requests if attrs will not be updated.

  • Enabled language support across APIs.

  • Gets random article if no arguments.

  • CLI scripts and tests disabled pending update.

0.0.4 (2016-08-16)

  • Added wptools.lead.

  • Added safe_exit() to CLI scripts.

  • Removed a fair amount of unused code.

0.0.3 (2016-08-12)

  • Implemented wptools.image choices.

  • Added wptools.api to simplify python i/f and CLI scripts.

  • Merged @0x9900’s CLI dist fixes.

  • A little more test coverage.

  • Starting to look like a legit module.

0.0.1 (2015)

  • Still better than alternatives for working with articles.

0.0.0 (2012)

  • It seems to work!

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

wptools-0.1.8.tar.gz (33.2 kB view hashes)

Uploaded Source

Built Distribution

wptools-0.1.8-py2.py3-none-any.whl (33.8 kB view hashes)

Uploaded Python 2 Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page