This is a Wikipedia Tool to fetch revisions based on a period of time.
Project description
Welcome to the Wikipedia Periodic Revisions (wikipedia_tools)
wikipedia_tools
is a Python Package to download wikipedia revisions for pages belonging to certain categories, based on a period of time. This package also provides overview stats for the downloaded data.
Dependencies and Credits
Wikipedia API
This package is built on top of the Wikipedia API. This code was forked under the base
subpackage.
ajoer/WikiRevParser
Also we forked the code from ajoer/WikiRevParser and we modified it to support from and to datetime to fetch revisions between certain periods; the modified code is wikipedia_toools.scraper.wikirevparser_with_time.py
.
Note: No need to download these two projects, they are already integrated as part of this project.
Installation
Via PIP
pip install wikipedia_tools
Or install manually by cloning and then running
pip install -e wikipedia_tools
wikipedia_tools package
This packages is responsible for:
- fetching the wikipages revisions based on a period of time
- load them into parquet, and
- provide basic analysis
It contains three main subpackages and the utils package which contains few helpers functions:
Scraper [wikipedia_tools.scraper]
This subpackage is responsible for downloading the wikipedia revisions from the web.
The code below shows how to download all the revisions of pages:
-
belonging to the Climate_change category.
-
revisions between start of 8 months ago (1.1.2022) and now (29.9.2022). The get_last_month function returns the datetime of the beginning of 8 months ago.
from wikipedia_tools.utils import utils utils.get_last_month(8)
-
if save_each_page= True: each page is fetched and downloaded on the spot under the folder data/periodic_wiki_batches/{categories_names}/from{month-year}_to{month-year}. Otherwise, all the page revisions are fetched first and then saved into one jsonl file.
from wikipedia_tools.scraper import downloader
from datetime import datetime
wikirevs= downloader.WikiPagesRevision(
categories = ["Climate_change"],
revisions_from = utils.get_last_month(8),
revisions_to=datetime.now(),
save_each_page= True
)
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for wikipedia_tools-2.1.9-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 542328d8032c78d31fad6799dd8cbfeda5714aead04c3c4fb561aac558658136 |
|
MD5 | 307b4cd513ab48132e31cdbeb32ae4e8 |
|
BLAKE2b-256 | 170ad8b3ae324beea8799c5a51138cdc96d5c67abc6cec434884e7a8802ff44a |