afaanoromo-tokenizer 0.1.2

Tokenizer library for Afaan Oromo supporting Unigram, BPE, and WordPiece algorithms.

Afaan Oromo Tokenizer

##Description Afan_oromo_tokenizer is a linguistically informed and computationally efficient tokenizer for Afaan Oromo, one of the most widely spoken low-resource languages in Africa.

-It can be used for different tasks such as:- -Neural Machine Translation (NMT) -Language Modelling -Text classification -... and so on.

##----background-----## -While tokenizers exist for major languages (English, Chinese, etc.), Afan Oromo lacks robust open-source tokenization tools.
-Afaan_oromo_tokenizer bridges that gap, facilitating NLP research for afaan Oromo.

A Python library providing tokenizers for the Afaan Oromo language using three popular subword algorithms: BPE, Unigram, and WordPiece. Ideal for NLP tasks.

---

Features

• Included: BPE, Unigram, WordPiece
• Trained on 13 million tokens
• Supports 425 unique tokens
• Vocabulary size for each tokenizer type: 55,000

---

Installation

pip install afaanoromo-tokenizer

Usage

from afaanoromo-tokenizer import ao_tokenizer

# Example text
text = "Afaanni oromoo afaan saba guddaati!"

# --- BPE tokenizer ---
bpe_tokenizer = ao_tokenizer("bpe")
bpe_tokens = bpe_tokenizer.encode(text)
print("BPE tokens:", bpe_tokens)
bpe_decoded = bpe_tokenizer.decode(bpe_tokens)
print("BPE decoded:", bpe_decoded)

# --- Unigram tokenizer ---
unigram_tokenizer = ao_tokenizer("unigram")
unigram_tokens = unigram_tokenizer.encode(text)
print("Unigram tokens:", unigram_tokens)
unigram_decoded = unigram_tokenizer.decode(unigram_tokens)
print("Unigram decoded:", unigram_decoded)

# --- WordPiece tokenizer ---
wordpiece_tokenizer = ao_tokenizer("wordpiece")
wordpiece_tokens = wordpiece_tokenizer.encode(text)
print("WordPiece tokens:", wordpiece_tokens)
wordpiece_decoded = wordpiece_tokenizer.decode(wordpiece_tokens)
print("WordPiece decoded:", wordpiece_decoded)

License

This project is licensed under the MIT License. See the LICENSE file for details.