Perplexity filter for documents and bulc HTML and WARC boilerplate removal.
Project description
Pyplexity
This package provides a simple interface to apply perplexity filters to any document. Furthermore, it provides a WARC and HTML bulk processor, with distributed capabilities.
Usage example
Process a folder containing a dataset using a trigrams model.
poetry build
pip3 install dist/pyplexity-0.1.31-py3-none-any.whl
pyplexity bulk-perplexity --perpl-model ../../clueweb-b13-rawtext2/trigrams_bnc.st --perpl-limit 8000.0 \
--trigrams --base-dir ./cleaned_webkb --output-dir ./perpl_filtered_webkb
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
pyplexity-0.1.35.tar.gz
(12.2 kB
view hashes)
Built Distribution
Close
Hashes for pyplexity-0.1.35-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c7a5793c3d65f433ebe3103a0655ee50726190dfb981a0bd21eb4f51762272c0 |
|
MD5 | 15690916bfb95df3147843c344267c62 |
|
BLAKE2b-256 | 02dc28958ea9e33968779fd232978c5fda154cfda50e5cb466f6dc78647d9cff |