bpe_knockout 2024.4.1

Implementation of BPE-knockout, a morphologically informed post-processing step for BPE tokenisers.

BPE-knockout

Repo hosting all the code used for the BPE-knockout paper. Below are the instructions for reproducing and extending the intrinsic evaluations. Extrinsic evaluations are done with RobBERT's framework.

Data

All data is included in the repo, because it is obtainable for free elsewhere and free of license too.

• Morphological decompositions were derived from WebCelex at the Max Plank Institute.
• Language modelling data is derived from OSCAR on HuggingFace.

Installing

Because this repo ships with data (word lists and tokenisers) that take quite a while to compute, we currently only guarantee that an editable install works. That is: you tell Python to use the folder into which you cloned the repo, rather than copying the code to your global or virtual site-packages directory.

git clone https://github.com/GitMew/BPE-knockout.git
cd BPE-knockout
pip install -r requirements.txt
pip install -e .

If you're using conda or venv, don't forget to activate those before running any calls to pip install.

Running

1. Unzip the .rar file under data/compressed/.
2. Run py main.py or python main.py in a terminal.

Using your own data

It is possible to use other datasets (even other languages) than the ones used for the paper. Here is how you would do that:

1. Make sure you have the following files:
   1. A word count file from a sufficiently large corpus;
   2. A file with morphological decompositions (not necessarily of the same words);
   3. Optional: if you don't want to generate a new BPE tokeniser from your word counts, the file(s) that specify your existing BPE tokeniser.
2. If your morphological decompositions are not in CELEX format, you still need to write your own parser for the morphology file. Do this in src/datahandlers/morphology.py by creating a subclass of the abstract LemmaMorphology class.
3. In src/auxiliary/config.py, create a new function that creates a ProjectConfig object declaring the paths to all the relevant files, as well as the name of the relevant LemmaMorphology subclass. Use the setup() functions as examples.
4. In main.py, specify this new config.