AistrighNLP - A collection of NLP tools for Irish
Project description
AistrighNLP
AistrighNLP is a collection of tools and models used for Aistrigh, the BT Young Scientist 2021 project. Our aim is to bring Irish into the modern era with NLP tools to give it parity with English. The tools included are based around the work in Neural Models for Predicting Celtic Mutations (Scannell, 2020). Included is all the tools needed to create a demutated Irish corpus, which can be used in all sorts of NLP tasks, and a model to reinsert them. For the Python API docs visit AistrighNLP Python API
Installing the Package
AistrighNLP can be downloaded using pip
pip install aistrigh-nlp
Lowercasing
When lowercasing either Irish or Scots Gaelic for prediciting mutations, you must be aware of special cases outlined in the paper above. Our lowercaser handles that
aistrigh-nlp lowercase -i input.txt -o output.txt
Removing mutations
To remove mutations from an entire dataset for use for NLP tasks (like Machine Translation) use demutate-corpus
aistrigh-nlp demutate-corpus -i input.txt -o output.txt
To remove mutations with a 'window' on either side to train a neural network, use demutate-window
, with -w
set to your desired window length on each side
aistrigh-nlp demutate-window -i input.txt -o output.csv -w 15
Predicting the mutations
To predict mutations on each word, use predict-mutations
. As of right now, it's only compatible with PyTorch+Torchtext/PyTorch models but we are working on expanding to TensorFlow and Keras. You'll need your vocab, labels and model checkpoint. We provide a default model to be used.
aistrigh-nlp predict-mutations -i input.txt -o output.txt -w 15 -v my_vocab.pth -l my_labels.pth -m my_model.pt
Applying the predicted mutations
To apply the mutations predicted by predict-mutations
, use apply-mutations
.
aistrigh-nlp apply-mutations -i input.txt -o output.txt -skip
-skip
here means to skip mutating any tokens containing symbols. It's unlikely these tokens need remutating, and will likely hurt performance without this set.
NOTE
AistrighNLP uses PyTorch Traces
to save the full computational graphs as checkpoints. This way, the model architecture need not be declared into hard-coded scripts. See this StackOverflow Thread for instructions to save a traced checkpoint.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for aistrigh_nlp-0.1.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5a21d9acb9be0ec7e57813959539611b285b905114b53ec76af5a8a2bf0e10bb |
|
MD5 | fa8b89153ff9d2d50433e36c8445387c |
|
BLAKE2b-256 | 31b7a54d799a4f7bc140fcbcbf08eeb7c0991425f6ee9e57351947ca899746b5 |