shobdo 0.2.2

A modern, high-performance Python library for Bengali dictionary operations.

Shobdo

A comprehensive Bengali dictionary and semantic search library for Python.

Installation · Quick Start · API Reference · Semantic Search · Contributing

---

Shobdo (শব্দ, "word") provides fast, type-safe access to a 45,630-word Bengali dictionary backed by SQLite and pre-computed neural embeddings. It is designed to replace slow, memory-heavy JSON-based dictionary loading with an indexed database that starts in milliseconds.

	JSON Loading	Shobdo
Startup	~1,200 ms	~6 ms
Memory	~180 MB	< 5 MB
Return type	dict	Pydantic Word

Features

• Exact, partial, and fuzzy search on Bengali headwords with configurable Levenshtein distance
• Reverse lookup from English translations to Bengali words
• Semantic search across all 45,630 entries using pre-computed 512-dimensional neural embeddings
• Cross-lingual queries — search in English, retrieve results in Bengali
• Synonym discovery via vector similarity, with no model inference at runtime
• Type-safe models — every result is a validated Pydantic Word object
• Zero configuration — all data ships with the package; just pip install and go

Installation

pip install shobdo

To enable semantic search capabilities (concept search, synonym finding):

pip install "shobdo[semantic]"

The semantic extra installs numpy and sentence-transformers. The base package requires only pydantic.

Quick Start

from shobdo import Shobdo

with Shobdo() as d:
    word = d.search("স্বাধীনতা")
    print(word.word)                # স্বাধীনতা
    print(word.pronunciation)       # শ্বাধীন্তা
    print(word.meanings)            # ['স্বাধীন হওয়ার ভাব', ...]
    print(word.english_translation) # Independence / Freedom
    print(word.part_of_speech)      # বিশেষ্য

Shobdo implements the context manager protocol. You can also instantiate it directly with d = Shobdo() and call d.close() when done.

API Reference

All methods are available on the Shobdo class.

search(word) -> Optional[Word]

Exact match lookup by Bengali headword. Returns None if not found.

result = d.search("অ")

lookup(query) -> List[Word]

Partial (substring) match on Bengali headwords.

words = d.lookup("ঋণ")
for w in words:
    print(w.word)
# ঋণ, ঋণগ্রস্ত, ঋণদাতা, ঋণপত্র, অঋণ, ...

search_fuzzy(query, max_distance=2) -> List[Word]

Fuzzy matching using Levenshtein edit distance. Useful for correcting user typos.

matches = d.search_fuzzy("পাকি", max_distance=1)
# Returns: [Word(word='পাখি', ...)]  — corrects the typo

search_english(query) -> List[Word]

Reverse lookup by English translation (substring match).

words = d.search_english("freedom")
for w in words:
    print(w.word, "-", w.english_translation)
# স্বাধীনতা - Independence / Freedom
# মুক্তি - Freedom / Release

get_random() -> Word

Returns a random dictionary entry.

word = d.get_random()
print(f"{word.word}: {word.meanings[0]}")

search_by_pos(pos, limit=50) -> List[Word]

Filter the dictionary by part of speech.

nouns = d.search_by_pos("বিশেষ্য", limit=5)
for w in nouns:
    print(w.word, "-", w.english_translation)

search_many(words) -> Dict[str, Optional[Word]]

Batch lookup of multiple words in a single query. Returns None for words not found.

results = d.search_many(["অ", "আনন্দ", "nonexistent"])
for word, entry in results.items():
    print(word, "→", entry.english_translation if entry else "not found")

stats() -> Dict

Returns a summary of the dictionary state.

d.stats()
# {'total_words': 45630, 'backend': 'sqlite', 'semantic_search': True}

---

Semantic Search

Shobdo ships with pre-computed 512-dimensional embeddings for every word in the dictionary. These vectors enable meaning-based retrieval: given a natural-language query, the library finds words that are conceptually related, even when they share no common characters.

The semantic extra is required: pip install "shobdo[semantic]".

search_semantic(query, top_k=10) -> List[Tuple[Word, float]]

Encode a free-text query at runtime and rank all dictionary entries by cosine similarity.

results = d.search_semantic("happiness and joy", top_k=5)

for word, score in results:
    print(f"{word.word}: {word.english_translation} ({score:.2f})")
# হর্ষোদয়: Rise of joy (19.86)
# আনন্দকন্দ: Root of joy (19.43)
# হর্ষাবিষ্ট: Overwhelmed with joy (19.26)
# সুখানুভব: Feeling of happiness (18.77)
# আনন্দ: Joy / Happiness (18.46)

Queries can be in English or Bengali. The model operates cross-lingually.

find_similar(word, top_k=10) -> List[Tuple[Word, float]]

Find semantically similar words (synonyms) using pre-computed embeddings. Because both sides of the comparison are pre-indexed, this method requires no model inference at runtime — it is a pure NumPy dot product.

synonyms = d.find_similar("আনন্দ", top_k=5)

for word, score in synonyms:
    print(f"{word.word}: {word.english_translation}")
# হর্ষোদয়: Rise of joy
# তোষ: Satisfaction / Pleasure
# আনন্দকন্দ: Root of joy
# হরষিত: Delighted / Joyful
# শর্ম: Happiness / Shelter

---

Data Models

All search results are returned as Pydantic models.

from shobdo.models import Word, Etymology

Word

Field	Type	Description
word	str	Bengali headword
pronunciation	Optional[str]	Phonetic pronunciation
part_of_speech	Optional[str]	Grammatical category
meanings	List[str]	Definitions
english_translation	Optional[str]	English equivalent
examples	List[str]	Usage examples
etymology	Optional[Etymology]	Origin information

Etymology

Field	Type	Description
source_language	Optional[str]	Source language (e.g., সংস্কৃত, আরবি)
derivation	Optional[str]	Morphological breakdown

---

Architecture

Shobdo separates data preparation (build-time) from data consumption (runtime) to keep the installed package fast and self-contained.

Build-time artifacts (shipped with the package):

File	Size	Contents
dictionary.db	~22 MB	SQLite database with indexed columns on word and english_translation
embeddings.npy	~45 MB	45,630 x 512 float16 embedding matrix
word_index.json	~3 MB	Maps vector indices to database row IDs

Runtime dependencies:

• Core — SQLite (stdlib) + Pydantic. No external services, no network calls.
• Semantic — adds NumPy (dot-product similarity) and sentence-transformers (query encoding).

---

Development

git clone https://github.com/inanXR/ProjectShobdo
cd library
pip install -e ".[semantic,dev]"

# Run the test suite
pytest tests/ -v

# Rebuild the SQLite database from source data
python scripts/build_db.py

# Regenerate embeddings (GPU recommended; see scripts/generate_embeddings_colab.ipynb)
python scripts/generate_embeddings.py

Requirements

Dependency	Required for
Python 3.9+	All
pydantic >= 2.0	All
numpy	Semantic search
sentence-transformers	Semantic search (query encoding)

License

Apache License 2.0

---

GitHub · Issues · PyPI

Built by Inan.