Sinlib

Sinlib Logo

A Python toolkit for Sinhala natural language processing — phonological tokenization, spell checking, and text preprocessing.

Note: The Romanizer and Transliterator modules are temporarily unavailable due to a known bug and will be restored in a future release.

Installation

pip install sinlib

Quick Start

Tokenization

from sinlib import Tokenizer

tokenizer = Tokenizer.from_pretrained("Ransaka/sinlib")

# Split into phonological units (base consonant + diacritics)
tokens = tokenizer.tokenize("ආයුබෝවන්")
# ['ආ', 'යු', 'බෝ', 'ව', 'න්']

# Encode to integer IDs
encoding = tokenizer("ආයුබෝවන්")
encoding.input_ids       # [4, 23, 18, 7, 12]
encoding.attention_mask  # [1, 1, 1, 1, 1]

# Batch encode with padding
batch = tokenizer(["ආයුබෝවන්", "සිංහල"], padding=True)
batch.input_ids  # [[4, 23, 18, 7, 12], [9, 31, 6, 0, 0]]

Spell Checking

from sinlib import TypoDetector

detector = TypoDetector.from_pretrained("Ransaka/sinlib")

# Auto-correct a sentence
detector("අපකරියට ගිය")
# 'අපකීර්තියට ගිය'

# Get correction suggestions
detector.suggest_correction("අඩිරාජ")
# ['අධිරාජ']

Neural Typo Correction (optional, CharBERT)

TypoDetector can optionally delegate hard cases to a Sinhala-CharBERT neural corrector — a dual-channel (subword + phonological akshara) seq2seq model. This catches noise classes the statistical dictionary pipeline cannot fix: Singlish transliteration, dialectal morphology (යන්ඩ → යන්න), ZWJ-damaged ligatures (ක්රීඩාව → ක්‍රීඩාව), split/fused words, and Unicode decomposition errors.

Install the optional dependency and pick a backend mode:

from sinlib import TypoDetector

# "denoise"  - bounded word-level neural fix when dictionary suggestions fail
# "seq2seq"  - open-vocabulary sentence-level fix when structural noise is detected
# "hybrid"   - both, in cascade (recommended)
detector = TypoDetector(neural_backend="hybrid")

detector("මම ගෙදර යන්ඩ ඕනේ")
# 'මම ගෙදර යන්න ඕනේ'

detector("මම gedara යන්න ඕනේ")
# 'මම ගෙදර යන්න ඕනේ'

detector("ක්රීඩාව")
# 'ක්‍රීඩාව'

Kwarg	Default	Description
`neural_backend`	`None`	`None`, `"denoise"`, `"seq2seq"`, or `"hybrid"`
`backend_model`	`Ransaka/sinhala-charbert-seq2seq`	HF Hub repo id or local checkpoint dir (`pytorch_model.bin` + `char_vocab.json`)
`backend_device`	auto (cuda > mps > cpu)	Torch device
`backend_revision`	`None`	Pin a Hub revision
`backend_num_beams`	`4`	Beam width for generation

Notes:

Default behavior is unchanged — without neural_backend the detector is purely statistical and requires no torch.
Neural corrections are gated: clean in-dictionary sentences never hit the model, and a candidate is accepted only if it scores no worse than the input (hallucination guard).
If the checkpoint cannot be downloaded, the detector degrades gracefully to statistical-only correction with a warning.

Preprocessing

from sinlib import preprocessing

# Remove noise and normalise text
clean = preprocessing.process_text("Hello, මේ සිංහල වාක්‍යකි.")

# Compute Sinhala character ratio
ratio = preprocessing.get_sinhala_character_ratio(["මෙය සිංහල වාක්‍යක්"])
# [0.9]

Why phonological tokenization?

Sinhala script combines a base consonant with one or more vowel diacritics into a single phonetic unit. Standard Unicode tokenization breaks these apart, producing incorrect representations for downstream tasks like ASR and TTS.

"ආයුබෝවන්"

Sinlib  →  ['ආ', 'යු', 'බෝ', 'ව', 'න්']   ✓ phonological units
Unicode →  ['ආ', 'ය', 'ු', 'බ', 'ෝ', 'ව', 'න', '්']   ✗ raw code points

Vocab and model weights are fetched automatically from Ransaka/sinlib on HuggingFace Hub at first use — no manual setup required.

Documentation

Full documentation is available at sinlib.readthedocs.io, including:

Contributing

Contributions are welcome. Please open an issue or submit a pull request on GitHub.

Fork the repository
Create a feature branch (git checkout -b feature/my-feature)
Commit your changes (git commit -m 'Add my feature')
Push to the branch (git push origin feature/my-feature)
Open a Pull Request

License

MIT License — see the LICENSE file for details.

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

sinlib-0.3.0.tar.gz (5.7 MB view details)

Uploaded Sep 1, 2026 Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

sinlib-0.3.0-py3-none-any.whl (5.0 MB view details)

Uploaded Sep 1, 2026 Python 3

File details

Details for the file sinlib-0.3.0.tar.gz.

File metadata

Download URL: sinlib-0.3.0.tar.gz
Upload date: Sep 1, 2026
Size: 5.7 MB
Tags: Source
Uploaded using Trusted Publishing? No
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for sinlib-0.3.0.tar.gz
Algorithm	Hash digest
SHA256	`3c1574a3ff2b773a928bf5aeacaf5c5f7ab03a65ceb0d2626321af4c32624aba`
MD5	`afa14ed5976a50630014ff10bbea4135`
BLAKE2b-256	`1d2bfe10c98876d572bb11a90219e00b56658daa5a3a5d2afc82dcaccb70ba5a`

See more details on using hashes here.

File details

Details for the file sinlib-0.3.0-py3-none-any.whl.

File metadata

Download URL: sinlib-0.3.0-py3-none-any.whl
Upload date: Sep 1, 2026
Size: 5.0 MB
Tags: Python 3
Uploaded using Trusted Publishing? No
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for sinlib-0.3.0-py3-none-any.whl
Algorithm	Hash digest
SHA256	`7a6780b4c3d0f5defefafc4a1dea0c6cfcce90acad46fd788bbb57326e320e0d`
MD5	`5319c8dfc719d559b43b36e4ac5b514b`
BLAKE2b-256	`4fe4bc652773f6fe3340a9ddbe1452772b9e463d573188e46ef9a3bd54b6cdb8`