bangla-text-toolkit

Zero-dependency Python library for Bangla (Bengali) NLP text preprocessing.

Built in public as a 12-week engineering roadmap. Each day adds one well-tested component to a growing pipeline — from raw Unicode to fixed-length token embeddings.

Components

Installation

pip install bangla-text-toolkit

Or install from source:

git clone https://github.com/Mouly22/bangla-text-toolkit.git
cd bangla-text-toolkit
pip install -e ".[dev]"

Quick start

from bangla_text_toolkit import (
    BanglaTextNormalizer,
    BanglaStemmer,
    BanglaVectorizer,
    BanglaKeywordExtractor,
    BanglaSequenceLabeler,
    BanglaEmbedder,
)
from bangla_text_toolkit.tokenizer import BanglaTokenizer

tok = BanglaTokenizer()
tokens = tok.tokenize("আমি বাংলায় গান গাই")

# Label tokens
labeler = BanglaSequenceLabeler()
print(labeler.label(tokens))
# -> [('আমি', 'STOP'), ('বাংলায়', 'STOP'), ('গান', 'WORD'), ('গাই', 'WORD')]

# Embed document as a fixed-length vector
emb = BanglaEmbedder(dim=64)
doc_vec = emb.embed_document(tokens)
print(len(doc_vec))   # 64

API reference

BanglaTextNormalizer

from bangla_text_toolkit import BanglaTextNormalizer
n = BanglaTextNormalizer(digit_mode="ascii")
n.normalize("আমি ০১২ বাংলা")  # -> "আমি 012 বাংলা"

BanglaTextCleaner

from bangla_text_toolkit.cleaner import BanglaTextCleaner
c = BanglaTextCleaner(remove_urls=True, remove_emojis=True)
c.clean("দেখো https://example.com 😊")  # -> "দেখো"

Pipeline

from bangla_text_toolkit.pipeline import Pipeline
pipe = Pipeline()
pipe.add_step(BanglaTextNormalizer().normalize)
result = pipe.run("  আমি বাংলা  ")

BanglaTokenizer

from bangla_text_toolkit.tokenizer import BanglaTokenizer, remove_stopwords
tok = BanglaTokenizer()
tok.tokenize("আমি বাংলায় গান গাই।")
# -> ["আমি", "বাংলায়", "গান", "গাই"]

BanglaRomanizer

from bangla_text_toolkit.romanization import BanglaRomanizer
r = BanglaRomanizer()
r.romanize("বাংলা")  # -> "bangla"

BanglaStemmer

from bangla_text_toolkit import BanglaStemmer
s = BanglaStemmer(min_stem_length=2)
s.stem("বাংলাদের")   # -> "বাংলা"
s.stem_tokens(["বাংলাদের", "গানগুলো"])  # -> ["বাংলা", "গান"]

BanglaVectorizer

from bangla_text_toolkit import BanglaVectorizer
corpus = [["আমি", "বাংলা"], ["সে", "বাংলা", "বলে"]]
vec = BanglaVectorizer(max_features=500, min_df=1, use_idf=True)
matrix = vec.fit_transform(corpus)
vec.get_feature_names()

BanglaKeywordExtractor

from bangla_text_toolkit import BanglaKeywordExtractor
corpus = [["আমি", "বাংলায়", "গান", "গাই"], ["সে", "বাংলায়", "কথা", "বলে"]]
kex = BanglaKeywordExtractor(top_k=3)
kex.fit(corpus)
kex.extract(corpus[0])
# -> [('গাই', 0.57...), ('গান', 0.40...), ('আমি', 0.40...)]

BanglaSequenceLabeler

from bangla_text_toolkit import BanglaSequenceLabeler
labeler = BanglaSequenceLabeler()
labeler.label(["আমি", "১২৩", "গান", "।"])
# -> [('আমি', 'STOP'), ('১২৩', 'NUM'), ('গান', 'WORD'), ('।', 'PUNCT')]

labeler.add_rule(r"[A-Za-z]+", "LATIN")  # custom rule, highest priority

BanglaEmbedder

from bangla_text_toolkit import BanglaEmbedder

emb = BanglaEmbedder(dim=64, ngram_range=(2, 4), normalize=True)

# Single token → 64-d L2-normalised vector
vec = emb.embed_token("বাংলা")

# Document → mean of token embeddings
doc_vec = emb.embed_document(["আমি", "বাংলায়", "গান", "গাই"])
print(len(doc_vec))  # 64

# Corpus → one vector per document
corpus_vecs = emb.embed_corpus([["আমি", "গান"], ["সে", "কথা"]])

Testing

pytest tests/ -v
# 246 tests, 0 failures, 0 dependencies

Roadmap

12-week build log — one component per session, all tested and CI-green.

Why this exists

Standard NLP tools silently break on Bangla text. Python's \w regex doesn't match Bangla combining vowel signs (Unicode category Mc/Mn), and most tokenisers treat matras as noise:

import re
re.findall(r'\w+', 'বাংলা')   # ['ব', 'ল']  ← drops 'া', 'ং'

This library handles the full Bangla Unicode block (U+0980–U+09FF) correctly, with no external dependencies.

License

MIT © Umme Abira Azmary