Bindings to a rust library that can help process wikipedia's data dumps.
Project description
Wikicleaner
Wikipedia provides a series of dumps in XML format. While developing this project, I personally used the simple wiki dumps.
Purpose
The intent of this project is to provide some low-level routines that allow us to manipulate the content of XML dumps. One such routine removes all of the annotations like [[File:*]] and {{some metadata}} in order to make our input more digestible for an AI application that uses embeddings.
Example Usage
>>> import wikicleaner as wc
>>> article_raw_text = "[[File:Colorful spring garden.jpg|thumb|180px|right|[[Spring]] flowers in April in the [[Northern Hemisphere]].]] April comes between [[March]] and [[May]]"
>>> wc.clean_article_text(article_raw_text)
' April comes between March and May'
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file wikicleaner-0.1.7.tar.gz.
File metadata
- Download URL: wikicleaner-0.1.7.tar.gz
- Upload date:
- Size: 24.2 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: maturin/1.4.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
ff8e7ebc77e692a1b26041e042e3f8159237e6f3f4ec86ce47c2fefbd24e8931
|
|
| MD5 |
6e9dbaf08d80656f2786a9cfece38c32
|
|
| BLAKE2b-256 |
1b02b6b10791f29ca743584835205c352f63d4d1717beed53e5d331ac3200bf2
|
File details
Details for the file wikicleaner-0.1.7-cp311-cp311-manylinux_2_34_x86_64.whl.
File metadata
- Download URL: wikicleaner-0.1.7-cp311-cp311-manylinux_2_34_x86_64.whl
- Upload date:
- Size: 825.3 kB
- Tags: CPython 3.11, manylinux: glibc 2.34+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: maturin/1.4.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
56d38be6a8a740bb0cb522b1dcc819b6594dc05d6345931846432e6f9ea8a3fa
|
|
| MD5 |
fa57c710c0e0ba155a0f5b05efd94399
|
|
| BLAKE2b-256 |
83bb686d1c655e752943f2db39e6421eaf2cb58a0a32ae2f8a6cfca4f8c5e7cf
|
