wikipedia_tools 2.1.7

This is a Wikipedia Tool to fetch revisions based on a period of time.

Welcome to the Wikipedia Periodic Revisions (wikipedia_tools)

wikipedia_tools is a Python Package to download wikipedia revisions for pages belonging to certain categories, based on a period of time. This package also provides overview stats for the downloaded data.

---

Dependencies and Credits

Wikipedia API

This package is built on top of the Wikipedia API. This code was forked under the base subpackage.

ajoer/WikiRevParser

Also we forked the code from ajoer/WikiRevParser and we modified it to support from and to datetime to fetch revisions between certain periods; the modified code is wikipedia_toools.scraper.wikirevparser_with_time.py.

Note: No need to download these two projects, they are already integrated as part of this project.

Installation

Via PIP

pip install wikipedia_tools

Or install manually by cloning and then running

pip install -e wikipedia_tools

wikipedia_tools package

This packages is responsible for:

• fetching the wikipages revisions based on a period of time
• load them into parquet, and
• provide basic analysis

It contains three main subpackages and the utils package which contains few helpers functions:

Scraper [wikipedia_tools.scraper]

This subpackage is responsible for downloading the wikipedia revisions from the web.

The code below shows how to download all the revisions of pages:

• belonging to the Climate_change category.

• revisions between start of 8 months ago (1.1.2022) and now (29.9.2022). The get_last_month function returns the datetime of the beginning of 8 months ago.

  from wikipedia_tools.utils import utils 
  utils.get_last_month(8)
  

• if save_each_page= True: each page is fetched and downloaded on the spot under the folder data/periodic_wiki_batches/{categories_names}/from{month-year}_to{month-year}. Otherwise, all the page revisions are fetched first and then saved into one jsonl file.

from wikipedia_tools.scraper import downloader
from datetime import datetime

wikirevs= downloader.WikiPagesRevision( 
                                        categories = ["Climate_change"],
                                        revisions_from = utils.get_last_month(8),
                                        revisions_to=datetime.now(),
                                        save_each_page= True
                                        )