Skip to main content

A python module to generate a tokenized dump of Wikipedia for NLP

Project description


GitHub release PyPI release Build MIT License

Welcome to WiToKit, a Python toolkit to download and generate preprocessed Wikipedia dumps for all languages.

WiToKit can be used to converts a Wikipedia archive into a single .txt file, one (tokenized) sentence per line.

Note: WiToKit currently only supports xx-pages-articles.xml.xx.bz2 Wikipedia archives corresponding to articles, templates, media/file descriptions, and primary meta-pages.


pip3 install witokit

On python3.5 you may need to pass on the --process-dependency-link flag:

pip3 install witokit --process-dependency-link



To download a .bz2-compressed Wikipedia XML dump, do:

witokit download ⁠\
  --lang lang_wp_code \
  --date wiki_date \
  --output /abs/path/to/output/dir/where/to/store/bz2/archives \
  --num-threads num_cpu_threads

For example, to download the latest English Wikipedia, do:

witokit download ⁠--lang en --date latest --output /abs/path/to/output/dir --num-threads 2

The --lang parameter expects the WP (language) code corresponding to the desired Wikipedia archive. Check out the full list of Wikipedias with their corresponding WP codes here.

The --date parameter expects a string corresponding to one of the dates found under the Wikimedia dump site corresponding to a given Wikipedia dump (e.g. for the English Wikipedia).

Important Keep num-threads <= 3 to avoid rejection from Wikimedia servers


To extract the content of the downloaded .bz2 archives, do:

witokit extract \
  --input /abs/path/to/downloaded/wikipedia/bz2/archives \
  --num-threads num_cpu_threads


To preprocess the content of the extracted XML archives and output a single .txt file, tokenize, one sentence per line:

witokit process \
  --input /abs/path/to/wikipedia/extracted/xml/archives \
  --output /abs/path/to/single/output/txt/file \
  --lower \  # if set, will lowercase text
  --num-threads num_cpu_threads

Preprocessing for all languages is performed with Polyglot.


You can also use WiToKit to sample the content of a preprocess .txt file, using:

witokit sample \
  --input /abs/path/to/witokit/preprocessed/txt/file \
  --percent \  # percentage of total lines to keep
  --balance  # if set, will balance sampling, otherwise, will take top n sentences only

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

witokit-1.1.0.tar.gz (8.2 kB view hashes)

Uploaded source

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page