shobdo 0.2.0

A modern, high-performance Python library for Bengali dictionary operations.

শব্দ | Shobdo

A high-performance Bengali dictionary library for Python

45,000+ words • SQLite backend • 512D neural embeddings • Pydantic models

---

---

Overview

Shobdo provides fast, type-safe access to a comprehensive Bengali dictionary. The data is pre-compiled into SQLite and indexed, eliminating the startup cost of loading JSON into memory.

Metric	JSON Loading	Shobdo
Startup time	~1,200ms	~6ms
Memory footprint	~180MB	<5MB
Return type	dict	Pydantic Word

---

Installation

pip install shobdo

For semantic search (optional):

pip install "shobdo[semantic]"

---

Quick Start

from shobdo import Shobdo

d = Shobdo()
word = d.search("স্বাধীনতা")

print(word.word)                # স্বাধীনতা
print(word.pronunciation)       # শ্বাধীন্‌তা
print(word.meanings)            # ['স্বাধীন হওয়ার ভাব', ...]
print(word.english_translation) # Independence / Freedom
print(word.part_of_speech)      # বিশেষ্য

---

API Reference

search(word: str) -> Optional[Word]

Exact match lookup.

word = d.search("অ")
# Returns Word object or None

---

search_fuzzy(query: str, max_distance: int = 2) -> List[Word]

Levenshtein distance-based fuzzy matching. Useful for handling typos.

# User types "অরিন" instead of "অঋণ"
matches = d.search_fuzzy("অরিন", max_distance=2)

for m in matches:
    print(m.word, m.english_translation)
# অঋণ Debt-free
# অরণ্য Forest

---

search_english(query: str) -> List[Word]

Search by English translation (substring match).

words = d.search_english("freedom")

for w in words:
    print(w.word, w.english_translation)
# স্বাধীনতা Independence / Freedom
# মুক্তি Freedom / Release

---

lookup(query: str) -> List[Word]

Partial match on Bengali headwords.

words = d.lookup("ঋণ")

for w in words[:5]:
    print(w.word)
# ঋণ
# ঋণগ্রস্ত
# ঋণদাতা
# ঋণপত্র
# অঋণ

---

get_random() -> Word

Returns a random word from the dictionary.

word = d.get_random()
print(f"{word.word}: {word.meanings[0]}")

---

stats() -> Dict

Returns dictionary statistics.

print(d.stats())
# {'total_words': 45630, 'backend': 'sqlite', 'semantic_search': True}

---

Semantic Search

Shobdo includes pre-computed 512-dimensional embeddings (via EmbeddingGemma-300m, truncated using MRL & stored in float16) for all words. This enables meaning-based search.

search_semantic(query: str, top_k: int = 10) -> List[Tuple[Word, float]]

Search by concept. The query is encoded at runtime and compared against all word embeddings.

results = d.search_semantic("happiness and joy", top_k=5)

for word, score in results:
    print(f"{word.word}: {word.english_translation} ({score:.2f})")
# হর্ষোদয়: Rise of joy (19.86)
# আনন্দকন্দ: Root of joy (19.43)
# হর্ষাবিষ্ট: Overwhelmed with joy (19.26)
# সুখানুভব: Feeling of happiness (18.77)
# আনন্দ: Joy / Happiness (18.46)

Works cross-lingually (English query → Bengali results).

---

find_similar(word: str, top_k: int = 10) -> List[Tuple[Word, float]]

Find synonyms using pre-computed embeddings. No model inference required at runtime.

synonyms = d.find_similar("আনন্দ", top_k=5)

for word, score in synonyms:
    print(f"{word.word}: {word.english_translation}")
# হর্ষোদয়: Rise of joy
# তোষ: Satisfaction / Pleasure
# আনন্দকন্দ: Root of joy
# হরষিত: Delighted / Joyful
# শর্ম: Happiness / Shelter

---

Data Models

from shobdo.models import Word, Etymology

Word

Field	Type	Description
word	str	Bengali headword
pronunciation	Optional[str]	Phonetic pronunciation
part_of_speech	Optional[str]	Grammatical category
meanings	List[str]	List of definitions
english_translation	Optional[str]	English equivalent
examples	List[str]	Usage examples
etymology	Optional[Etymology]	Word origin

Etymology

Field	Type	Description
source_language	Optional[str]	e.g., সংস্কৃত, আরবি
derivation	Optional[str]	Morphological breakdown

---

Architecture

Build-time artifacts:

• dictionary.db (~15MB) — SQLite database with indexed word and english_translation columns
• embeddings.npy (~46MB) — 45,630 × 512 float16 array
• word_index.json (~3MB) — Vector index to database ID mapping

Runtime:

• SQLite queries for exact/fuzzy/partial/reverse search
• NumPy dot product for semantic similarity

---

Development

git clone https://github.com/inanXR/ProjectShobdo
cd Shobdo-Library
pip install -e ".[dev]"

# Run tests
pytest tests/

# Rebuild database
python scripts/build_db.py

# Regenerate embeddings (~10 min)
python scripts/generate_embeddings.py

---

Requirements

• Python 3.9+
• Core: pydantic
• Semantic search: numpy, sentence-transformers

---

License

Apache License 2.0

---

Acknowledgments

Data source: ProjectShobdo

---

GitHub · Issues · PyPI