MediaWiki Streaming
A set of utilities for stream-processing MediaWiki data.
Usage
mwstream (-h | --help)
mwstream <utility> [-h|--help]
Data processing utilities
- diffs2persistence
Generates token persistence statistics using revision JSON blobs with diff information.
- dump2json
Converts an XML dump to a stream of revision JSON blobs
- dump2diffs
Computes diffs directly from an XML dump
- json2diffs
Computes and adds a “diff” field to a stream of revision JSON blobs
- mend_diffs
Mends diffs that were computed in chunks and out of order.
- persistence2stats
Aggregates a token persistence statistics to revision statistics
- wikihadoop2json
Converts a Wikihadoop-processed stream of XML pages to JSON blobs
General utilities
- json2tsv
Converts a stream of JSON blobs to tab-separated values based a set of fieldnames.
- normalize
Normalizes old versions of RevisionDocument json schemas to correspond to the most recent schema version.
- validate
Validates JSON against a provided schema.
- truncate_text
Truncates the ‘text’ field of JSON blobs to a limited length in unicode characters. (addresses content dump vandalism issues) and adds a boolean ‘truncated’ field.
Installation
pip install mwstreaming
2 maintainers
Metadata
2 maintainers
Release files for mwstreaming 0.5.5
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distributions (sdists)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| mwstreaming-0.5.5.zip | 23.3 kB | Without provenance | Details for mwstreaming-0.5.5.zip | |
| mwstreaming-0.5.5.tar.gz | 12.5 kB | Without provenance | Details for mwstreaming-0.5.5.tar.gz |
Release files / mwstreaming-0.5.5.zip
| Download URL | mwstreaming-0.5.5.zip |
|---|---|
| Size | 23.3 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
69e02a0764edd40fbd9786e984e19c35e9c2aaa55c08ab8454ac245effeb5bb0
|
|
BLAKE2b-256 checksum How to use checksums |
8140c6239436534240c056e0f97fd866e69701cdd8633081ea7714da0e0262a7
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
Release files / mwstreaming-0.5.5.tar.gz
| Download URL | mwstreaming-0.5.5.tar.gz |
|---|---|
| Size | 12.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
7d4544b8ac8237e24373e047347c77efee7d675f0cd1db558326c99d4b6344f3
|
|
BLAKE2b-256 checksum How to use checksums |
194a7be116169b1105a993d93beced9bd5759268bba8c429e9c27a3e0b43f7a0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
Security for mwstreaming 0.5.5
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
