bangla-nlpkit

A lightweight, zero-dependency Python toolkit for Bangla (Bengali) NLP — with support for Chakma, Myanmar (Marma), Devanagari, Arabic, and Latin scripts.

---

Features

Module	Functions	Description
detector	detect_script, is_bangla	Unicode-range script detection across 6 script families
normalizer	normalize, to_ascii_digits, to_bangla_digits, remove_invisible	Unicode cleanup, ZWJ/ZWNJ fix, digit conversion, whitespace normalization
tokenizer	word_tokenize, sentence_tokenize, char_tokenize	Bangla-aware word and sentence splitting with danda (।) support
romanizer	romanize, romanize_word, romanize_tokens	Bangla script → Roman transliteration (NLC scheme), with verified correct handling of word-final inherent-vowel suppression and decomposed Unicode nukta sequences (ড়/ঢ়/য়) — see benchmarks

Zero external dependencies — pure Python standard library only.

---

Installation

pip install bangla-nlpkit

---

Quick Start

from bangla_nlpkit import detect_script, normalize, word_tokenize, romanize

# Script detection
result = detect_script("আমি বাংলায় কথা বলি")
print(result.dominant)          # bangla
print(result.scores)            # {'bangla': 1.0, 'devanagari': 0.0, 'arabic': 0.0, 'latin': 0.0, 'chakma': 0.0, 'myanmar': 0.0}

# Mixed script
result = detect_script("আমি English মিশিয়ে বলি")
print(result.dominant)          # bangla
print(result.is_mixed())        # True

# Normalization
text = normalize("  আমি  বাংলায়  কথা  বলি  ")
print(text)                     # আমি বাংলায় কথা বলি

# Digit conversion
print(normalize("আমার বয়স ২৩", normalize_digits=True))   # আমার বয়স 23

# Tokenization
tokens = word_tokenize("আমি বাংলায় কথা বলি।")
print(tokens)   # ['আমি', 'বাংলায়', 'কথা', 'বলি']

sents = sentence_tokenize("আমি বাংলায় কথা বলি। তুমি কি বাংলায় কথা বলো?")
print(sents)    # ['আমি বাংলায় কথা বলি।', 'তুমি কি বাংলায় কথা বলো?']

# Romanization
print(romanize("বাংলাদেশ"))     # bangladesh
print(romanize("ঢাকা"))         # dhaka
print(romanize("মুক্তিযুদ্ধ")) # muktijuddh

---

Supported Scripts

Script	Languages	Unicode Range
Bangla	Bengali, Sylheti	U+0980–U+09FF
Latin	English, Garo	Basic Latin + Extended
Arabic	Arabic, Urdu	U+0600–U+06FF
Devanagari	Hindi, Sanskrit	U+0900–U+097F
Chakma	Chakma	U+11100–U+1114F
Myanmar	Marma (Burmese)	U+1000–U+109F

Note: Garo language uses Latin script and will be detected as latin. For language-level disambiguation between Garo and English, a language classifier model is required.

---

API Reference

detect_script(text) → DetectionResult

result = detect_script("আমি বাংলায় কথা বলি")
result.dominant          # 'bangla'
result.scores            # {'bangla': 1.0, 'latin': 0.0, ...}
result.char_counts       # {'bangla': 18, 'latin': 0, ...}
result.total_classified  # 18
result.is_mixed()        # False

normalize(text, *, nfc, remove_zwj, normalize_digits, to_ascii_digits, normalize_punctuation, strip) → str

normalize("  আমি  বাংলায়  ")              # 'আমি বাংলায়'
normalize("আমার ২৩", normalize_digits=True) # 'আমার 23'
normalize("\u201Chello\u201D")              # '"hello"'

word_tokenize(text, *, keep_punctuation) → list[str]

word_tokenize("আমি বাংলায় কথা বলি।")                    # ['আমি', 'বাংলায়', 'কথা', 'বলি']
word_tokenize("আমি বলি।", keep_punctuation=True)         # ['আমি', 'বলি', '।']

sentence_tokenize(text) → list[str]

sentence_tokenize("বাক্য এক। বাক্য দুই।")
# ['বাক্য এক।', 'বাক্য দুই।']

romanize(text, *, passthrough_non_bangla) → str

romanize("বাংলাদেশ")                           # 'bangladesh'
romanize("আমি English মিশিয়ে বলি")            # 'ami English mishiye boli'
romanize("আমি English বলি", passthrough_non_bangla=False)  # 'amiboli'

Note: with passthrough_non_bangla=False, ALL non-Bangla characters are dropped, including whitespace and punctuation surrounding non-Bangla segments — the output is not re-spaced afterward.

---

Verified Correctness — Decomposed Unicode Nukta Handling

Bengali characters ড়, ঢ়, and য় can appear in Unicode text either as a single precomposed code point or as a base consonant followed by a combining nukta mark (U+09BC). These two forms are not unified by standard Unicode NFC normalization (they are on the Unicode Composition Exclusion Table), so any tool relying on NFC normalization to treat them identically will fail on one form or the other.

We benchmarked bangla-nlpkit against indic_transliteration (a maintained Python package implementing ISO 15919) on 40 curated Bangla words containing these characters in word-initial, medial, and final position:

System	Precomposed = Decomposed output match	Leaked nukta in output
bangla-nlpkit	40/40 (100%)	0/40 (0%)
indic_transliteration (ISO 15919)	0/40 (0%)	40/40 (100%)

Full benchmark code and results: benchmarks/benchmark_nukta.py. This result has been independently reproduced across two platforms (Linux, Windows).

---

Running Tests

pip install bangla-nlpkit[dev]
pytest tests/ -v

---

Background

This library was developed alongside research on low-resource Bangladeshi languages (Bengali, Chakma, Garo, Marma) published at ITAI 2026 (Springer LNNS, Lasell University, USA). It addresses practical preprocessing gaps encountered when working with mixed-script parallel corpora.

---

Contributing

Issues and pull requests are welcome. See CONTRIBUTING.md for guidelines.

---

License

MIT — see LICENSE.

---

Author

Rashedul Hasan Shohan
BSc CSE, Daffodil International University, Bangladesh
GitHub · Google Scholar · ORCID