meranaw-tokenizer 0.1.4

A Meranaw tokenizer library trained specifically on translated religious texts (Bible and Quran).

Meranaw Tokenizer

A Python library for tokenizing the Meranaw language using subword segmentation models (BPE and Unigram). These models were trained only on translated Meranaw religious texts (Quran and Bible). This library was developed as part of a thesis study to facilitate the development of Meranaw NLP applications.

Features

• Two Algorithms: Supports Byte Pair Encoding (BPE) and Unigram Language Model.
• Three Sizes: Includes Small, Medium, and Large vocabulary models.
• Simple API: Easy encoding and decoding functions.

Installation

pip install meranaw-tokenizer

Quick Start

import meranaw_tokenizer as mt

# 1. Load a model (Default: Large Unigram)
tokenizer = mt.load_model()

# 2. Tokenize text
text = "Mapiya kapipita" 
text_lower = text.lower() 
tokens = tokenizer.encode(text_lower, out_type=str)
ids = tokenizer.encode(text_lower, out_type=int)

print(f"Tokens: {tokens}")
print(f"IDs: {ids}")

# 3. Decode back to text
decoded_text = tokenizer.decode(ids)
print(f"Decoded: {decoded_text}")

Available Model Sizes

Size	Vocab Count
small	2,315
medium	6,402
large	19,859

Usage Notes

• Normalization: These models were trained on normalized, lowercase Meranaw text. To maintain low subword fertility (preventing unnecessary fragmentation), always .lower() your string before encoding.
• Algorithms:
  • Unigram: Uses a probabilistic approach to tokenization.
  • BPE: Uses a frequency-based merging approach.

Author

Jamail M. Cali
MSU Main - Marawi City
Email: [e-mail]

License

This project is licensed under the MIT License - see the LICENSE file for details.