Toolbox for filtering parallel corpora
Project description
OpusFilter
OpusFilter is a tool for filtering and combining parallel corpora.
Features:
- Corpus preprocessing pipelines configured with YAML
- Simple downloading of parallel corpora from OPUS with OpusTools
- Implementations for many common text file operations on parallel files
- Memory-efficient processing of large files
- Implemented filters based e.g. on language identification, word aligment, n-gram language models, and multilingual sentence embeddings
- Extendable with your own filters written in Python
OpusFilter has been presented in ACL 2020 system demonstrations.
Installing
Install the latest release from PyPI:
pip install opusfilter
orpip install opusfilter[all]
(include optional Python libraries)
Install from source:
pip install .
orpython setup.py install
Documentation
The complete OpusFilter documentation is available from helsinki-nlp.github.io/OpusFilter.
You can also build the documents from the source:
pip install -r docs/requirements.txt
orpip install .[docs]
sphinx-build docs docs-html
Changelog
A changelog is available in docs/CHANGELOG.md.
Citing
If you use OpusFilter in your research, please cite our ACL 2020 paper:
@inproceedings{aulamo-etal-2020-opusfilter,
title = "{O}pus{F}ilter: A Configurable Parallel Corpus Filtering Toolbox",
author = {Aulamo, Mikko and Virpioja, Sami and Tiedemann, J{\"o}rg},
booktitle = "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations",
month = jul,
year = "2020",
publisher = "Association for Computational Linguistics",
url = "https://www.aclweb.org/anthology/2020.acl-demos.20",
doi = "10.18653/v1/2020.acl-demos.20",
pages = "150--156"
}
A full bibliography of papers cited in the documentation and code can be found from docs/references.bib.
Contributing
See docs/CONTRIBUTING.md.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
opusfilter-2.5.1.tar.gz
(103.1 kB
view hashes)
Built Distribution
opusfilter-2.5.1-py3-none-any.whl
(51.2 kB
view hashes)
Close
Hashes for opusfilter-2.5.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | a859a9fee2ea0bf5506a0c77f66d60b3a08fafd493c8ac390c90ad4b242f1806 |
|
MD5 | 39a7b4c15464271bd1e88c3228f61f3b |
|
BLAKE2b-256 | dbc5e1bacfbd3670af9f45d50b6834ff59f5c4cba8dd5c15a7a4d59529766161 |