Usage

Install the package using pip:

pip install wiki_dump_parser

Then, use it directly from command line:

python -m wiki_dump_parser <dump.xml>

Or from python code:

import wiki_dump_parser as parser
parser.xml_to_csv('dump.xml')

The output csv files should be loaded using ‘|’ as an escape character for quoting string. An example to load the output file “dump.csv” generated by this script using pandas would be:

df = pd.read_csv('dump.csv', quotechar='|', index_col = False)
df['timestamp'] = pd.to_datetime(df['timestamp'],format='%Y-%m-%dT%H:%M:%SZ')

Dependencies

• python 3

Yes, nothing more.

How to get a wiki history dump

There are several ways to get the wiki dump:

• If you have access to the server, follow the instructions in the mediawiki docs.

• For Wikia wikis and many other domains, you can use our in-house developed script made to accomplish this task. It is straightforward to use and very fast on it.

• Wikimedia project wikis: For wikis belonging to the Wikimedia project, you already have a regular updated repo with all the dumps here: http://dumps.wikimedia.org. Select your target wiki from the list and download the complete edit history dump and uncompress it.

• For other wikis, like self-hosted wikis, you should use the wikiteam’s dumpgenerator.py script. You have a simple tutorial in their wiki. Its usage is very straightforward and the script is well maintained. Remember to use the –xml option to download the full history dump.