Sinlib
A Python toolkit for Sinhala natural language processing — phonological tokenization, spell checking, and text preprocessing.
Note: The
RomanizerandTransliteratormodules are temporarily unavailable due to a known bug and will be restored in a future release.
Installation
pip install sinlib
Quick Start
Tokenization
from sinlib import Tokenizer
tokenizer = Tokenizer.from_pretrained("Ransaka/sinlib")
# Split into phonological units (base consonant + diacritics)
tokens = tokenizer.tokenize("ආයුබෝවන්")
# ['ආ', 'යු', 'බෝ', 'ව', 'න්']
# Encode to integer IDs
encoding = tokenizer("ආයුබෝවන්")
encoding.input_ids # [4, 23, 18, 7, 12]
encoding.attention_mask # [1, 1, 1, 1, 1]
# Batch encode with padding
batch = tokenizer(["ආයුබෝවන්", "සිංහල"], padding=True)
batch.input_ids # [[4, 23, 18, 7, 12], [9, 31, 6, 0, 0]]
Spell Checking
from sinlib import TypoDetector
detector = TypoDetector.from_pretrained("Ransaka/sinlib")
# Auto-correct a sentence
detector("අපකරියට ගිය")
# 'අපකීර්තියට ගිය'
# Get correction suggestions
detector.suggest_correction("අඩිරාජ")
# ['අධිරාජ']
Neural Typo Correction (optional, CharBERT)
TypoDetector can optionally delegate hard cases to a Sinhala-CharBERT neural
corrector — a dual-channel (subword + phonological akshara) seq2seq model. This
catches noise classes the statistical dictionary pipeline cannot fix: Singlish
transliteration, dialectal morphology (යන්ඩ → යන්න), ZWJ-damaged ligatures
(ක්රීඩාව → ක්රීඩාව), split/fused words, and Unicode decomposition errors.
Install the optional dependency and pick a backend mode:
from sinlib import TypoDetector
# "denoise" - bounded word-level neural fix when dictionary suggestions fail
# "seq2seq" - open-vocabulary sentence-level fix when structural noise is detected
# "hybrid" - both, in cascade (recommended)
detector = TypoDetector(neural_backend="hybrid")
detector("මම ගෙදර යන්ඩ ඕනේ")
# 'මම ගෙදර යන්න ඕනේ'
detector("මම gedara යන්න ඕනේ")
# 'මම ගෙදර යන්න ඕනේ'
detector("ක්රීඩාව")
# 'ක්රීඩාව'
| Kwarg | Default | Description |
|---|---|---|
neural_backend |
None |
None, "denoise", "seq2seq", or "hybrid" |
backend_model |
Ransaka/sinhala-charbert-seq2seq |
HF Hub repo id or local checkpoint dir (pytorch_model.bin + char_vocab.json) |
backend_device |
auto (cuda > mps > cpu) | Torch device |
backend_revision |
None |
Pin a Hub revision |
backend_num_beams |
4 |
Beam width for generation |
Notes:
- Default behavior is unchanged — without
neural_backendthe detector is purely statistical and requires no torch. - A neural candidate is accepted only if it scores no worse than the input (hallucination guard), so already-clean text is never degraded.
- If the checkpoint cannot be downloaded, the detector degrades gracefully to statistical-only correction with a warning.
Preprocessing
from sinlib import preprocessing
# Remove noise and normalise text
clean = preprocessing.process_text("Hello, මේ සිංහල වාක්යකි.")
# Compute Sinhala character ratio
ratio = preprocessing.get_sinhala_character_ratio(["මෙය සිංහල වාක්යක්"])
# [0.9]
Why phonological tokenization?
Sinhala script combines a base consonant with one or more vowel diacritics into a single phonetic unit. Standard Unicode tokenization breaks these apart, producing incorrect representations for downstream tasks like ASR and TTS.
"ආයුබෝවන්"
Sinlib → ['ආ', 'යු', 'බෝ', 'ව', 'න්'] ✓ phonological units
Unicode → ['ආ', 'ය', 'ු', 'බ', 'ෝ', 'ව', 'න', '්'] ✗ raw code points
Vocab and model weights are fetched automatically from Ransaka/sinlib on HuggingFace Hub at first use — no manual setup required.
Documentation
Full documentation is available at sinlib.readthedocs.io, including:
Contributing
Contributions are welcome. Please open an issue or submit a pull request on GitHub.
- Fork the repository
- Create a feature branch (
git checkout -b feature/my-feature) - Commit your changes (
git commit -m 'Add my feature') - Push to the branch (
git push origin feature/my-feature) - Open a Pull Request
License
MIT License — see the LICENSE file for details.
1 maintainer
Release files for sinlib 0.3.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| sinlib-0.3.2.tar.gz | 5.7 MB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| sinlib-0.3.2-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 10.7 MB
Release files / sinlib-0.3.2.tar.gz
| Download URL | sinlib-0.3.2.tar.gz |
|---|---|
| Size | 5.7 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
7084bbeb0a7139457594d5b0d2d3f262e60bb07807caf27e1578c5af77f7874b
|
|
BLAKE2b-256 checksum How to use checksums |
d6715ca5cba75a275fa6892ba22f985b84a14565209e42492ed06f7e1bc158ec
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Release files / sinlib-0.3.2-py3-none-any.whl
| Download URL | sinlib-0.3.2-py3-none-any.whl |
|---|---|
| Size | 5.0 MB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
bf9fd4a51cfccc118f06f997e02ede8354e114a291fa93c348b93ec413976803
|
|
BLAKE2b-256 checksum How to use checksums |
a2bc167a419b24451bb9d571d11bd07717207e53b89ee5280eb25a24a26a303b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
