Sinlib
A Python toolkit for Sinhala natural language processing — phonological tokenization, spell checking, and text preprocessing.
Note: The
RomanizerandTransliteratormodules are temporarily unavailable due to a known bug and will be restored in a future release.
Installation
pip install sinlib
Quick Start
Tokenization
from sinlib import Tokenizer
tokenizer = Tokenizer.from_pretrained("Ransaka/sinlib")
# Split into phonological units (base consonant + diacritics)
tokens = tokenizer.tokenize("ආයුබෝවන්")
# ['ආ', 'යු', 'බෝ', 'ව', 'න්']
# Encode to integer IDs
encoding = tokenizer("ආයුබෝවන්")
encoding.input_ids # [4, 23, 18, 7, 12]
encoding.attention_mask # [1, 1, 1, 1, 1]
# Batch encode with padding
batch = tokenizer(["ආයුබෝවන්", "සිංහල"], padding=True)
batch.input_ids # [[4, 23, 18, 7, 12], [9, 31, 6, 0, 0]]
Spell Checking
from sinlib import TypoDetector
detector = TypoDetector.from_pretrained("Ransaka/sinlib")
# Auto-correct a sentence
detector("අපකරියට ගිය")
# 'අපකීර්තියට ගිය'
# Get correction suggestions
detector.suggest_correction("අඩිරාජ")
# ['අධිරාජ']
Neural Typo Correction (optional, CharBERT)
TypoDetector can optionally delegate hard cases to a Sinhala-CharBERT neural
corrector — a dual-channel (subword + phonological akshara) seq2seq model. This
catches noise classes the statistical dictionary pipeline cannot fix: Singlish
transliteration, dialectal morphology (යන්ඩ → යන්න), ZWJ-damaged ligatures
(ක්රීඩාව → ක්රීඩාව), split/fused words, and Unicode decomposition errors.
Install the optional dependency and pick a backend mode:
from sinlib import TypoDetector
# "denoise" - bounded word-level neural fix when dictionary suggestions fail
# "seq2seq" - open-vocabulary sentence-level fix when structural noise is detected
# "hybrid" - both, in cascade (recommended)
detector = TypoDetector(neural_backend="hybrid")
detector("මම ගෙදර යන්ඩ ඕනේ")
# 'මම ගෙදර යන්න ඕනේ'
detector("මම gedara යන්න ඕනේ")
# 'මම ගෙදර යන්න ඕනේ'
detector("ක්රීඩාව")
# 'ක්රීඩාව'
| Kwarg | Default | Description |
|---|---|---|
neural_backend |
None |
None, "denoise", "seq2seq", or "hybrid" |
backend_model |
Ransaka/sinhala-charbert-seq2seq |
HF Hub repo id or local checkpoint dir (pytorch_model.bin + char_vocab.json) |
backend_device |
auto (cuda > mps > cpu) | Torch device |
backend_revision |
None |
Pin a Hub revision |
backend_num_beams |
4 |
Beam width for generation |
Notes:
- Default behavior is unchanged — without
neural_backendthe detector is purely statistical and requires no torch. - A neural candidate is accepted only if it scores no worse than the input (hallucination guard), so already-clean text is never degraded.
- If the checkpoint cannot be downloaded, the detector degrades gracefully to statistical-only correction with a warning.
Preprocessing
from sinlib import preprocessing
# Remove noise and normalise text
clean = preprocessing.process_text("Hello, මේ සිංහල වාක්යකි.")
# Compute Sinhala character ratio
ratio = preprocessing.get_sinhala_character_ratio(["මෙය සිංහල වාක්යක්"])
# [0.9]
Why phonological tokenization?
Sinhala script combines a base consonant with one or more vowel diacritics into a single phonetic unit. Standard Unicode tokenization breaks these apart, producing incorrect representations for downstream tasks like ASR and TTS.
"ආයුබෝවන්"
Sinlib → ['ආ', 'යු', 'බෝ', 'ව', 'න්'] ✓ phonological units
Unicode → ['ආ', 'ය', 'ු', 'බ', 'ෝ', 'ව', 'න', '්'] ✗ raw code points
Vocab and model weights are fetched automatically from Ransaka/sinlib on HuggingFace Hub at first use — no manual setup required.
Documentation
Full documentation is available at sinlib.readthedocs.io, including:
Contributing
Contributions are welcome. Please open an issue or submit a pull request on GitHub.
- Fork the repository
- Create a feature branch (
git checkout -b feature/my-feature) - Commit your changes (
git commit -m 'Add my feature') - Push to the branch (
git push origin feature/my-feature) - Open a Pull Request
License
MIT License — see the LICENSE file for details.
1 maintainer
Metadata
1 maintainer
Release files for sinlib 0.3.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | |
|---|---|---|---|---|
| sinlib-0.3.2.tar.gz | 5.7 MB | Without provenance | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| sinlib-0.3.2-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 10.7 MB
Release files / sinlib-0.3.2.tar.gz
| Download URL | sinlib-0.3.2.tar.gz |
|---|---|
| Size | 5.7 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
7084bbeb0a7139457594d5b0d2d3f262e60bb07807caf27e1578c5af77f7874b
|
|
BLAKE2b-256 checksum How to use checksums |
d6715ca5cba75a275fa6892ba22f985b84a14565209e42492ed06f7e1bc158ec
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Release files / sinlib-0.3.2-py3-none-any.whl
| Download URL | sinlib-0.3.2-py3-none-any.whl |
|---|---|
| Size | 5.0 MB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
bf9fd4a51cfccc118f06f997e02ede8354e114a291fa93c348b93ec413976803
|
|
BLAKE2b-256 checksum How to use checksums |
a2bc167a419b24451bb9d571d11bd07717207e53b89ee5280eb25a24a26a303b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/7.0.0 CPython/3.13.14
|
Security for sinlib 0.3.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.
