Sinlib
A Python toolkit for Sinhala natural language processing — phonological tokenization, spell checking, and text preprocessing.
Note: The
RomanizerandTransliteratormodules are temporarily unavailable due to a known bug and will be restored in a future release.
Installation
pip install sinlib
Quick Start
Tokenization
from sinlib import Tokenizer
tokenizer = Tokenizer.from_pretrained("Ransaka/sinlib")
# Split into phonological units (base consonant + diacritics)
tokens = tokenizer.tokenize("ආයුබෝවන්")
# ['ආ', 'යු', 'බෝ', 'ව', 'න්']
# Encode to integer IDs
encoding = tokenizer("ආයුබෝවන්")
encoding.input_ids # [4, 23, 18, 7, 12]
encoding.attention_mask # [1, 1, 1, 1, 1]
# Batch encode with padding
batch = tokenizer(["ආයුබෝවන්", "සිංහල"], padding=True)
batch.input_ids # [[4, 23, 18, 7, 12], [9, 31, 6, 0, 0]]
Spell Checking
from sinlib import TypoDetector
detector = TypoDetector.from_pretrained("Ransaka/sinlib")
# Auto-correct a sentence
detector("අපකරියට ගිය")
# 'අපකීර්තියට ගිය'
# Get correction suggestions
detector.suggest_correction("අඩිරාජ")
# ['අධිරාජ']
Preprocessing
from sinlib import preprocessing
# Remove noise and normalise text
clean = preprocessing.process_text("Hello, මේ සිංහල වාක්යකි.")
# Compute Sinhala character ratio
ratio = preprocessing.get_sinhala_character_ratio(["මෙය සිංහල වාක්යක්"])
# [0.9]
Why phonological tokenization?
Sinhala script combines a base consonant with one or more vowel diacritics into a single phonetic unit. Standard Unicode tokenization breaks these apart, producing incorrect representations for downstream tasks like ASR and TTS.
"ආයුබෝවන්"
Sinlib → ['ආ', 'යු', 'බෝ', 'ව', 'න්'] ✓ phonological units
Unicode → ['ආ', 'ය', 'ු', 'බ', 'ෝ', 'ව', 'න', '්'] ✗ raw code points
Vocab and model weights are fetched automatically from Ransaka/sinlib on HuggingFace Hub at first use — no manual setup required.
Documentation
Full documentation is available at sinlib.readthedocs.io, including:
Contributing
Contributions are welcome. Please open an issue or submit a pull request on GitHub.
- Fork the repository
- Create a feature branch (
git checkout -b feature/my-feature) - Commit your changes (
git commit -m 'Add my feature') - Push to the branch (
git push origin feature/my-feature) - Open a Pull Request
License
MIT License — see the LICENSE file for details.
1 maintainer
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file sinlib-0.2.4.tar.gz.
File metadata
- Download URL: sinlib-0.2.4.tar.gz
- Upload date:
- Size: 5.7 MB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
7a65a10009ab49eff9614ce878a1ae881fb8310bb97b4cb13401185a4428556b
|
|
| MD5 |
92ccbf35b9885ec23f05c6d4bbec1274
|
|
| BLAKE2b-256 |
7baa7bd5272a02058c3e16b1e6f68fc4717ee386cc028d4fc2a0b2d5cf96a5af
|
File details
Details for the file sinlib-0.2.4-py3-none-any.whl.
File metadata
- Download URL: sinlib-0.2.4-py3-none-any.whl
- Upload date:
- Size: 5.0 MB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via:
twine/7.0.0 CPython/3.13.14
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
d10a27517bedc383934a54bbf75729b9380f05344e56ae31a89a3a952f6180aa
|
|
| MD5 |
ee91485b00c4f4b37e40800ce29f0bc3
|
|
| BLAKE2b-256 |
e88fc204de88ae49bea9d800571e7516914bf2049561ffed18f0bcbb4cd2fcd0
|
