Perplexity filter for documents and bulc HTML and WARC boilerplate removal.
Project description
Pyplexity
This package provides a simple interface to apply perplexity filters to any document. Furthermore, it provides a WARC and HTML bulk processor, with distributed capabilities.
Usage example
Process a folder containing a dataset using a trigrams model.
poetry build
pip3 install dist/pyplexity-0.1.31-py3-none-any.whl
pyplexity bulk-perplexity --perpl-model ../../clueweb-b13-rawtext2/trigrams_bnc.st --perpl-limit 8000.0 \
--trigrams --base-dir ./cleaned_webkb --output-dir ./perpl_filtered_webkb
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
pyplexity-0.1.31.tar.gz
(12.2 kB
view hashes)
Built Distribution
Close
Hashes for pyplexity-0.1.31-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 02315ab33b1e8cf16d57cba60669890adbccc6cf5f20554c63f6b35cb76fd651 |
|
MD5 | f1c23b6ee2c54db8843cff59fed7b0a9 |
|
BLAKE2b-256 | caaa4e725dde70df629b5b229061572fd6298994fab8af7c0f8996fa242da2ba |