MediaWiki Streaming
A set of utilities for stream-processing MediaWiki data.
Usage
mwstream (-h | --help)
mwstream <utility> [-h|--help]
Data processing utilities
- diffs2persistence
Generates token persistence statistics using revision JSON blobs with diff information.
- dump2json
Converts an XML dump to a stream of revision JSON blobs
- json2diffs
Computes and adds a “diff” field to a stream of revision JSON blobs
- persistence2stats
Aggregates a token persistence statistics to revision statistics
- wikihadoop2json
Converts a Wikihadoop-processed stream of XML pages to JSON blobs
General utilities
- json2tsv
Converts a stream of JSON blobs to tab-separated values based a set of fieldnames.
- normalize
Normalizes old versions of RevisionDocument json schemas to correspond to the most recent schema version.
- validate
Validates JSON against a provided schema.
- truncate_text
Truncates the ‘text’ field of JSON blobs to a limited length in unicode characters. (addresses content dump vandalism issues) and adds a boolean ‘truncated’ field.
Installation
pip install mwstreaming
2 maintainers
Release files for mwstreaming 0.5.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distributions (sdists)
| File | Size | Uploaded | |
|---|---|---|---|
| mwstreaming-0.5.0.zip | 19.6 kB | Details | |
| mwstreaming-0.5.0.tar.gz | 10.7 kB | Details |
Release files / mwstreaming-0.5.0.zip
| Download URL | mwstreaming-0.5.0.zip |
|---|---|
| Size | 19.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
f4b2bc578766aa8f74edef5b1be60af80a5b2225b9dcc2e10ed5c08f8ae41afb
|
|
BLAKE2b-256 checksum How to use checksums |
3c79e02468c03532922b6c0ba82c4987a9d43dfc0123981785554f62e12675eb
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
Release files / mwstreaming-0.5.0.tar.gz
| Download URL | mwstreaming-0.5.0.tar.gz |
|---|---|
| Size | 10.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
c4fef8ffb86ee873ef07d98d0090c37851f0e3f73b4cd4839d91088be1b47b20
|
|
BLAKE2b-256 checksum How to use checksums |
e8f986b557963de3b26d698521577548cc942cfb0f43b7dbce196810e505b5c0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
