Skip to main content

OpusFilter

OpusFilter is a tool for filtering and combining parallel corpora.

Features:

  • Corpus preprocessing pipelines configured with YAML
  • Simple downloading of parallel corpora from OPUS with OpusTools
  • Implementations for many common text file operations on parallel files
  • Memory-efficient processing of large files
  • Implemented filters based e.g. on language identification, word aligment, n-gram language models, and multilingual sentence embeddings
  • Extendable with your own filters written in Python

OpusFilter has been presented in ACL 2020 system demonstrations.

Installing

Install the latest release from PyPI:

  • pip install opusfilter or pip install opusfilter[all] (include optional Python libraries)

Install from source:

  • pip install . or python setup.py install

Troubleshooting

OpusFilter should generally work fine on Python 3.9 to 3.14. In the case of troubles, try installing the exact versions in requirements.txt:

  • pip install -r requirements.txt

Documentation

The complete OpusFilter documentation is available from helsinki-nlp.github.io/OpusFilter.

You can also build the documents from the source:

  • pip install -r docs/requirements.txt or pip install .[docs]
  • sphinx-build docs docs-html

Changelog

A changelog is available in docs/CHANGELOG.md.

Citing

If you use OpusFilter in your research, please cite our ACL 2020 paper:

@inproceedings{aulamo-etal-2020-opusfilter,
    title = "{O}pus{F}ilter: A Configurable Parallel Corpus Filtering Toolbox",
    author = {Aulamo, Mikko and Virpioja, Sami and Tiedemann, J{\"o}rg},
    booktitle = "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations",
    month = jul,
    year = "2020",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.acl-demos.20",
    doi = "10.18653/v1/2020.acl-demos.20",
    pages = "150--156"
}

A full bibliography of papers cited in the documentation and code can be found from docs/references.bib.

Contributing

See docs/CONTRIBUTING.md.

Acknowledgements

OpusFilter has partially been supported by various projects such as:

  • MeMAD - Methods for Managing Audiovisual Data (EU Horizon 2020)
  • FoTran - Found in Translation (EU ERC)
  • HPLT - High-Performance Language Technologies (EU Horizon)
  • GreenNLP - Controlling the Carbon Footprint in Sustainable Language Technology (Academy of Finland ICT 2023)

Metadata

Release files for opusfilter 3.4.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for opusfilter 3.4.0
File Size Uploaded Provenance Actions
opusfilter-3.4.0.tar.gz 185.6 kB Without provenance Details for opusfilter-3.4.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for opusfilter 3.4.0
File Interpreter ABI Platform Actions
opusfilter-3.4.0-py3-none-any.whl Python 3 none any Details for opusfilter-3.4.0-py3-none-any.whl

Total release size: 284.4 kB

Release files / opusfilter-3.4.0.tar.gz

Download URL opusfilter-3.4.0.tar.gz
Size 185.6 kB
Tags Source
SHA-256 checksum
How to use checksums
4ce25ca1e45f00f0e1d6e3e1ec29b638fbb454da3ed78e7f5ff82d1ee0d5b81c
BLAKE2b-256 checksum
How to use checksums
c7b5b9b58c2e81d5c77822c0aa2d905c01b40c71e70fa12785817dc6002ad2a9
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.16

Release files / opusfilter-3.4.0-py3-none-any.whl

Download URL opusfilter-3.4.0-py3-none-any.whl
Size 98.8 kB
Tags Python 3
SHA-256 checksum
How to use checksums
a1b841f097cc985e54dd4995a85c46824cb5ba4ed7bbbb825656f7b7e5c1cc26
BLAKE2b-256 checksum
How to use checksums
799c71fa868810b106dba95f78118113d72f7e2d3261464d800a4d7b3f5adbb4
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/7.0.0 CPython/3.13.16

Security for opusfilter 3.4.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page