Skip to main content

Wikipedia tools (for Humans)

Project description

Wikipedia tools (for Humans)

.. image::

.. image::

Python and command-line MediaWiki access for Humans.

- get an HTML or plain text "extract" (lead or summary)
- get a representative image (pageimage, thumb, etc.)
- get an Infobox as a python dictionary
- get any/all Wikidata by title
- get info in any language
- get random info

This package is intended to make it as easy as possible to get data
from MediaWiki instances, expose more Wikidata, and extend the
MediaWiki API just for kicks. We say "(for Humans)" because that is a
goal_. See our wiki_ to learn more. Questions, feedback, and
especially contributions_ are welcome!

.. _NOTES:
.. _contributions:
.. _goal:
.. _wiki:

- Install_
- Usage_
- Examples_
- Methods_
- wptool_

.. _Install:


.. code-block:: bash

$ pip install wptools

.. _Usage:


.. code-block:: python

>>> import wptools

An instance can be initialized by:

**** (self)

- ``None``: <NoneType>
- ``lang``: <str> MediaWiki `language code`_ (default='en')
- ``pageid``: <int> a MediaWiki Page ID
- ``title``: <str> a MediaWiki article title
- ``wiki``: <str> any MediaWiki site
- ``wikibase``: <str> Wikidata item `entity ID`_

.. _`language code`:
.. _`entity ID`:

The simplest way to begin is with a title:

.. code-block:: python

>>> f ="Flannery O'Connor")
Flannery_O'Connor (en)
lang: en
title: Flannery_O'Connor

The default language is 'en' (English):

.. code-block:: python

>>> t ='穐吉敏子')
穐吉敏子 (en)
lang: en
title: 穐吉敏子

Leaving off arguments invokes a random_ lookup in English:

.. code-block:: python

>>> r = (random) None
Sylvia_Rivera (en)
lang: en
pageid: 3296309
title: Sylvia_Rivera

.. _random:

If you give only *lang*, you get a random_ article in that language:

.. code-block:: python

>>> zh ='zh') (random) None
哈莉特·塔布曼 (zh)
lang: zh
pageid: 211070
title: 哈莉特·塔布曼

You can also start with a *wikibase* item:

.. code-block:: python

>>> q ='Q43303')
Q43303 (en)
lang: en
wikibase: Q43303

Or, another *wiki* site:

.. code-block:: python

>>> m ='') (random) None
Malala_Yousafzai (en)
lang: en
pageid: 146817
title: Malala_Yousafzai

Instance attributes echo automatically. You can turn that off with

.. code-block:: python

>>> r =

Request details echo to *stderr* with ``verbose=True``:

.. code-block:: python

>>> r =

All API entrypoints support setting ``proxy`` and ``timeout`` (in seconds):

.. code-block:: python

>>> r.get(proxy='', timeout=5)

.. _Examples:


Get a representative image:

.. code-block:: python

>>> frida ='Frida Kahlo').get_query() (query) Frida_Kahlo (imageinfo) File:Frida Kahlo, by Guillermo Kahlo.jpg|Fi...

>>> frida.image('page')['url']

>>> frida.image('thumb')['url']


.. image::

**Note**: A page's image can come from the ``pageimage`` or
``thumbnail`` (via ``get_query()``), from an Infobox (via
``get_parse()``), from Wikidata Property:P18_ (via
``get_wikidata()``), or from the RESTBase ``image`` or ``thumb`` (via
``get_rest()``). All images are stored in the ``images`` attribute,
and can be queried with the ``image()`` method.

Get a text (or HTML) extract:

.. code-block:: python

>>> ella ='Ella Fitzgerald').get_query() (query) Ella_Fitzgerald (imageinfo) File:Ella Fitzgerald (Gottlieb 02871).jpg|F...

>>> print ella.extext
**Ella Jane Fitzgerald** (April 25, 1917 – June 15, 1996) was an
American jazz singer often referred to as the First Lady of Song,
Queen of Jazz and Lady Ella. She was noted for her purity of tone,
impeccable diction, phrasing and intonation, and a "horn-like"
improvisational ability, particularly in her scat singing.

>>> print ella.extract
<p><b>Ella Jane Fitzgerald</b> (April 25, 1917 – June 15, 1996) was an
American jazz singer often referred to as the First Lady of Song,
Queen of Jazz and Lady Ella. She was noted for her purity of tone,
impeccable diction, phrasing and intonation, and a "horn-like"
improvisational ability, particularly in her scat singing.</p>

Get an Infobox_ as a python object:

.. code-block:: python

>>> fela ='Fela Kuti').get_parse() (parse) Fela_Kuti (imageinfo) File:Fela Kuti.jpg

>>> fela.infobox['instrument']
'Saxophone, vocals, keyboards, trumpet, guitar, drums'

**Note**: Getting data from Infoboxen may be unavoidable, but getting
Wikidata (via ``get_wikidata()``) is preferred. Wikidata is
structured_ but (sometimes) data poor, while Infoboxen are
unstructured and (frequently) data rich. Please consider updating_
Wikidata if the information you want is only available in a MediaWiki
instance so that others may benefit from open, `linked data`_.

.. _structured:
.. _updating:
.. _`linked data`:

Get an (album, book, film, etc.) cover image:

.. code-block:: python

>>> blue ='Blue Train (album)').get_parse() (parse) Blue_Train_(album) (imageinfo) File:John Coltrane - Blue Train.jpg

>>> blue.image('cover')['url']


.. image::

Get wikidata by *title*:

.. code-block:: python

>>> fry ='Stephen Fry').get_wikidata() (wikidata) Stephen_Fry (claims) Q8817795|Q5|Q7066|Q145 (imageinfo) File:Stephen Fry cropped.jpg
Stephen_Fry (en)
cache: <dict(2)> {claims, wikidata}
claims: <dict(4)> {Q145, Q5, Q7066, Q8817795}
description: English comedian, actor, writer, presenter, and activist
images: <dict(1)> {wikidata-image}
label: Stephen Fry
lang: en
modified: <dict(1)> {wikidata}
props: <dict(8)> {P135, P18, P27, P31, P345, P569, P856, P910}
title: Stephen_Fry
what: human
wikibase: Q192912
wikidata: <dict(8)> {IMDB, birth, category, citizenship, image, in...

**Note**: Resolved properties and claims are stored in the
``wikidata`` attribute. Wikidata properties are selected by
``_WIKIPROPS``. Properties (e.g. P17_ "country") are stored in
``props`` and those properties that have Wikidata items for values
(e.g. Q142_ for "France") are stored in ``claims`` and resolved by
another Wikidata API call (as shown above). See the `Wikidata page`_
in our wiki for more details.

.. _P17:
.. _Q142:
.. _`Wikidata page`:

Extend Wikidata claims_ to be resolved:

.. code-block:: python

>>> simone ='Simone de Beauvoir')
>>> simone._WIKIPROPS['P21'] = 'gender'
>>> simone.get_wikidata() (wikidata) Simone_de_Beauvoir (claims) Q142|Q5|Q3411417|Q859773|Q151578|Q1214721|Q470... (imageinfo) File:Simone de Beauvoir.jpg

>>> simone.wikidata['gender']

.. _claims:

Get special (experimental) `lead section`_ HTML:

.. code-block:: python

>>> buddha ='Buddha').get_rest() (/page/mobile-text/) Buddha (imageinfo) File:Buddha in Sarnath Museum (Dhammajak Mutra).jpg

>>> buddha.lead
<img query-thumbnail src="
<span heading><a href="">Gau...
<span snipped><span><b>Gautama Buddha</b>, also known as <b>Siddhārtha G...
Gautama taught a <a href="" titl...
Gautama is the primary figure in Buddhism. He is recognized by Buddhists...
<span metadata>Modified: 2016-10-13T09:44:13Z</span>

**Note**: The *lead* attribute contains an assembled stand-alone,
encyclopedia-like HTML fragment:

- ``<img {kind}>`` selected image
- ``<span heading>`` wiki-linked title and description
- ``<span snipped>`` lead paragraphs with (noprint, reference, &c.) snipped
- ``<span metadata>`` available metadata (e.g. Last modified)

Get all the things:

.. code-block:: python

>>> jill ='Jill Lepore').get() (query) Jill_Lepore (parse) 22469182 (wikidata) Q6192915 (claims) Q30|Q5
Jill_Lepore (en)
cache: <dict(4)> {claims, parse, query, wikidata}
claims: <dict(2)> {Q30, Q5}
description: American historian
extext: <str(1016)> **Jill Lepore** (born August 27, 1966) is an A...
extract: <str(1114)> <p><b>Jill Lepore</b> (born August 27, 1966) ...
infobox: <dict(38)> {academic_advisors, alma_mater, alt, author_ab...
label: Jill Lepore
lang: en
modified: <dict(2)> {page, wikidata}
pageid: 22469182
parsetree: <str(50677)> <root><template><title>Infobox scientist</...
props: <dict(3)> {P27, P31, P569}
random: Ramesh Bidhuri
title: Jill_Lepore
what: human
wikibase: Q6192915
wikidata: <dict(3)> {birth, citizenship, instance}
wikitext: <str(22540)> {{Infobox scientist| name = Jill Lepore| na...

Query results are cached in the ``cache`` attribute:

.. code-block:: python

>>> jill.cache['query']['info']
{'bytes': 1911.0,
'content': 'application/json; charset=utf-8',
'kB/s': '2.6',
'seconds': '0.743',
'status': 200,
'url': '|url|watchers&list=random&pithumbsize=240&ppprop=wikibase_item&prop=extracts|images|info|pageimages|pageprops&redirects&rnlimit=1&rnnamespace=0&titles=Jill_Lepore',
'user-agent': 'wptools/0.1.7 ( PycURL/7.43.0 libcurl/7.49.1 SecureTransport zlib/1.2.8'}

The ``wptools`` user-agent_ will look like this:


wptools/*version* ( *libs*

.. _user-agent:

.. _Methods:


Get help on instance methods like this:

.. code-block:: python

>>> help(wptools.core)
>>> help(<instance>)

**get** (self)

make all requests necessary to populate all the things, probably:

- get_query()
- get_parse()
- get_wikidata()

**get_claims** (self)

Wikidata:API `action=wbgetentities`_ for labels of claims

- e.g. {'Q298': 'country'} resolves to {'country': 'Chile'}
- use get_wikidata() to populate claims

**get_imageinfo** (self)

MediaWiki request for `API:Imageinfo`_

- images: <dict> updates image URLs, sizes, etc.

.. _`API:Imageinfo`:

**get_parse** (self)

MediaWiki:API `action=parse`_ request for:

- images: <dict> {parse-image, parse-cover}
- infobox: <dict> Infobox_ data as python dictionary
- links: <list> interwiki links (iwlinks_)
- pageid: <int> MediaWiki database ID
- parsetree: <str> `XML parse tree`_
- wikibase: <str> Wikidata `entity ID`_ or wikidata URL
- wikitext: <str> raw wikitext URL

.. _Infobox:
.. _`XML parse tree`:
.. _`action=parse`:
.. _iwlinks:

**get_query** (self)

MediaWiki:API `action=query`_ request for:

- description: <str> Wikidata description (via pageterms)
- extext: <str> plain text (Markdown_) extract
- extract: <str> HTML extract via `Extension:TextExtract`_
- images: <dict> {query-pageimage, query-thumbnail}
- label: <str> Wikidata label (via pageterms)
- modified (page): <str> ISO8601 date and time
- pageid: <int> MediaWiki database ID
- random: <str> a random article title with every request!
- url: <str> the canonical wiki URL
- url_raw: <str> ostensible raw wikitext URL

.. _Markdown:
.. _`Extension:TextExtract`:
.. _`action=query`:

**get_random** (self)

MediaWiki:API `action=query`_ request for:

- pageid: <int> MediaWiki database ID
- title: <str> article title

**get_rest** (self)

RESTBase_ ``/page/mobile-text/`` request for:

- description: <str> apparently, Wikidata description
- images: <dict> {rest-image, rest-thumb}
- lead: <str> encyclopedia-like `lead section`_
- modified (page): <str> ISO8601 date and time
- url: <str> the canonical wiki URL
- url_raw: <str> ostensible raw wikitext URL

.. _`lead section`:
.. _RESTBase:

**get_wikidata** (self)

Wikidata:API `action=wbgetentities`_ request for:

- claims: <dict> Wikidata claims (to be resolved)
- description: <str> Wikidata description
- images: <dict> {wikidata-image} Wikidata Property:P18_
- label: <str> Wikidata label
- modified (wikidata): <str> ISO8601 date and time
- props: <dict> Wikidata properties
- what: <str> Wikidata Property:P31_ "instance of"
- wikibase: <str> Wikidata item ID
- wikidata: <dict> resolved Wikidata properties
- wikidata_url: <str> Wikidata URL

.. _Property:P18:
.. _Property:P31:
.. _`action=wbgetentities`:

**image** (self, token)

Returns first image info with kind containing token (e.g. 'thumb')

**show** (self)

Pretty-print instance attributes.

.. _wptool:


Basic functionality on the command-line is provided by the ``wptool`` command.

.. code-block:: bash

$ wptool -h
usage: wptool [-h] [-H] [-l L] [-n] [-q] [-s] [-t T] [-v] [-w W]

Get Wikipedia article info and Wikidata via MediaWiki APIs.

Gets a random English Wikipedia article by default, or in the
language -lang, or from the wikisite -wiki, or by specific
title -title. The output is a plain text extract unless -HTML.

optional arguments:
-h, --help show this help message and exit
-H, -HTML output HTML extract
-l L, -lang L language code
-n, -nowrap do not wrap text
-q, -query show query and exit
-s, -shh quiet output to stderr
-t T, -title T get a specific title
-v, -verbose HTTP status to stderr
-w W, -wiki W use alternative wikisite

Powered by

For example:

.. code-block:: bash

$ wptool -t "Jeanne d'Arc" -l fr -s
JEANNE_D'ARC—sainte et héroïne de l'histoire de France

![Jeanne d'Arc](

**Jeanne d'Arc**, née vers 1412 à Domrémy village du duché de Bar dont
une partie relevait du royaume de France pour le temporel et de
l'évêché de Toul pour le spirituel (actuellement dans le département
des Vosges en Lorraine), et morte sur le bûcher le 30 mai 1431 à
Rouen, capitale du duché de Normandie alors possession du royaume
d'Angleterre, est une héroïne de l'histoire de France, chef de guerre
et sainte de l'Église catholique, surnommée depuis le XVIe siècle «
_la Pucelle d'Orléans_ » et, depuis le XIXe siècle, « _mère de la
nation française_ ».


Please enjoy!

@siznax 👹

.. :changelog:

Release History

0.1.10 (2016-11-06)

* Many fixes/improvements with help of @seansay (thank you!)
* Raise LookupError on API missing page/title
* Consolidated images and caches
* Support get_query() by pageid
* Support proxy, timeout on all API entrypoints
* Support for language variants (e.g. variant='zh-cn')
* Support for non-https wikis via wiki=''
* Automatically get_imageinfo() via MediaWiki API:Imageinfo
* moved to wiki

0.1.7 (2016-09-20)

* Minor fixes
* Provide local SSL certs with python-certifi
* Enabled extending Wikidata properties
* Improved cache checking

0.1.4 (2016-09-06)

* Python 3 support!
* Implemented get_claims()
* Patch get_rest() path, tests/test_advanced
* Implemented get_wikidata() by title, lang
* Geo coordinates Property:P625 from Wikidata
* Added RESTBase support, lead attribute
* Made the atomic object

0.1.0 (2016-08-25)

* Many fixes, enhancements.
* Much testing of random (title, lang, and wiki).
* Allow chaining get_s or just get().
* Enabled full-hydration from wikibase only.
* Block further requests on fatal error.
* Get 240x thumbnails instead of default 50x.
* Harmonized silent and verbose arguments.
* Mock-up tests/
* Made a knowledge base.
* Use _humans_ in README examples. ;-)

0.0.5 (2016-08-23)

* Major re-write.
* Exposed core.WPTools as entrypoint.
* Added get_parse(), get_query(), and get_wikidata().
* Added get(self) to query all APIs.
* Added show(self) method to display fetched attrs.
* Show instance attributes after each request.
* Ignore requests if attrs will not be updated.
* Enabled language support across APIs.
* Gets random article if no arguments.
* CLI scripts and tests disabled pending update.

0.0.4 (2016-08-16)

* Added wptools.lead.
* Added safe_exit() to CLI scripts.
* Removed a fair amount of unused code.

0.0.3 (2016-08-12)

* Implemented wptools.image choices.
* Added wptools.api to simplify python i/f and CLI scripts.
* Merged @0x9900's CLI dist fixes.
* A little more test coverage.
* Starting to look like a legit module.

0.0.1 (2015)

* Still better than alternatives for working with articles.

0.0.0 (2012)

* It seems to work!

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for wptools, version 0.1.10
Filename, size File type Python version Upload date Hashes
Filename, size wptools-0.1.10-py2.py3-none-any.whl (34.2 kB) File type Wheel Python version 2.7 Upload date Hashes View hashes
Filename, size wptools-0.1.10.tar.gz (33.7 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page