Skip to main content

No project description provided

Project description

OpusCleaner

OpusCleaner is a machine translation/language model data cleaner and training scheduler. The Training scheduler has moved to OpusTrainer.

Cleaner

The cleaner bit takes care of downloading and cleaning multiple different datasets and preparing them for translation.

opuscleaner-clean --parallel 4 data/train-parts/dataset.filter.json | gzip -c > clean.gz

Installation for cleaning

If you just want to use OpusCleaner for cleaning, you can install it from PyPI, and then run it

pip3 install opuscleaner
opuscleaner-server serve

Then you can go to http://127.0.0.1:8000/ to show the interface.

You can also install and run OpusCleaner on a remote machine, and use SSH local forwarding (e.g. ssh -L 8000:localhost:8000 you@remote.machine) to access the interface on your local machine.

Dependencies

(Mainly listed as shortcuts to documentation)

  • FastAPI as the base for the backend part.
  • Pydantic for conversion of untyped JSON to typed objects. And because FastAPI automatically supports it and gives you useful error messages if you mess up things.
  • Vue for frontend

Screenshots

List and categorize the datasets you are going to use for training.

Download more datasets right from the interface.

Filter each individual dataset, showing you the results immediately.

Compare the dataset at different stages of filtering to see what the impact is of each filter.

Using your own data

OpusCleaner scans for datasets and finds them automatically if they're in the right format. When you download OPUS data, it will get converted to this format, and there's nothing stopping you from adding your own in the same format.

By default, it scans for files matching data/train-parts/*.*.gz and will derive which files make up a dataset from the filenames: name.en.gz and name.de.gz will be a dataset called name. The files are your standard moses format: a single sentence per line, and each Nth line in the first file will match with the Nth line of the second file.

When in doubt, just download one of the OPUS datasets through OpusCleaner, and replicate the format for your own dataset.

If you want to use another path, you can use the DATA_PATH environment variable to change it, e.g. run DATA_PATH="./my-datasets/*.*.gz" opuscleaner-server.

Paths

  • data/train-parts is scanned for datasets. You can change this by setting the DATA_PATH environment variable, the default is data/train-parts/*.*.gz.
  • filters should contain filter json files. You can change the FILTER_PATH environment variable, the default is <PYTHON_PACKAGE>/filters/*.json.

Installation for development

cd frontend
npm clean-install
npm run build
cd ..

python3 -m venv .env
bash --init-file .env/bin/activate
pip install -e .

Finally you can run opuscleaner-server as normal. The --reload option will cause it to restart when any of the python files change.

opuscleaner-server serve --reload

Then go to http://127.0.0.1:8000/ for the "interface" or http://127.0.0.1:8000/docs for the API.

Frontend development

If you're doing frontend development, try also running:

cd frontend
npm run dev

Then go to http://127.0.0.1:5173/ for the "interface".

This will put vite in hot-reloading mode for easier Javascript dev. All API requests will be proxied to the python server running in 8000, which is why you need to run both at the same time.

Filters

If you want to use LASER, you will also need to download its assets:

python -m laserembeddings download-models

Packaging

Run npm build in the frontend/ directory first, and then run hatch build . in the project directory to build the wheel and source distribution.

To push a new release to Pypi from Github, tag a commit with a vX.Y.Z version number (including the v prefix). Then publish a release on Github. This should trigger a workflow that pushes a sdist + wheel to pypi.

Acknowledgements

This project has received funding from the European Union’s Horizon Europe research and innovation programme under grant agreement No 101070350 and from UK Research and Innovation (UKRI) under the UK government’s Horizon Europe funding guarantee [grant number 10052546]

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

opuscleaner-0.4.2.tar.gz (307.5 kB view details)

Uploaded Source

Built Distribution

opuscleaner-0.4.2-py3-none-any.whl (341.3 kB view details)

Uploaded Python 3

File details

Details for the file opuscleaner-0.4.2.tar.gz.

File metadata

  • Download URL: opuscleaner-0.4.2.tar.gz
  • Upload date:
  • Size: 307.5 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.9.19

File hashes

Hashes for opuscleaner-0.4.2.tar.gz
Algorithm Hash digest
SHA256 305130cb6faa9aa5a033fb53b9a6db8697f3772ac19df93e153348566067712b
MD5 eb76731f2c7371ecb38bd71cb2a76c40
BLAKE2b-256 1496482b9cd2844dcecf2ce82cd469d7d9deb2b03608c3c4378331fabe97fb0b

See more details on using hashes here.

File details

Details for the file opuscleaner-0.4.2-py3-none-any.whl.

File metadata

  • Download URL: opuscleaner-0.4.2-py3-none-any.whl
  • Upload date:
  • Size: 341.3 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/5.1.1 CPython/3.9.19

File hashes

Hashes for opuscleaner-0.4.2-py3-none-any.whl
Algorithm Hash digest
SHA256 1365b86d8c336f7505cb9f4d6864f71fddde46449f1bce17a577a9333a0d7324
MD5 83b61cf3c95044f7590ad165aec51b1f
BLAKE2b-256 8f41b920a12aab9dc3359471f05991bcf38280fd33ee8987c1dfb3d4872ccb19

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page