sanskrit-tokenizer

Tokenize Sanskrit text with sandhi splitting for Information Retrieval.

pip install sanskrit-tokenizer

Quick start

from sanskrit_tokenizer import tokenize

tokenize("dharmakṣetre kurukṣetre")
# ['dharmakṣetre', 'kurukṣetre']

tokenize("धर्म योग")
# ['धर्म', 'योग']

tokenize("devālaya namaḥ")
# ['deva', 'ālaya', 'namaḥ']

tokenize() accepts both Devanagari and IAST input — output script matches input. Splits on whitespace and punctuation, applies reverse sandhi rules, and preserves the original script.

Sandhi splitting

from sanskrit_tokenizer.sandhi import split_sandhi

split_sandhi("devālaya")     # deva(832) + ālaya(76) — both in dict ✓
# ['deva', 'ālaya']

split_sandhi("narottama")    # nara(187) + uttama(160) — guṇa: a + u → o
# ['nara', 'uttama']

split_sandhi("dharmakṣetre")  # kṣa(0) + itre(0) — not real words ✗
# ['dharmakṣetre']

Dictionary-aware scoring: splits are validated against an embedded 8,750-word Sanskrit frequency dict (from Wiktionary, CC BY-SA). Candidates producing real words score dramatically higher than spurious fragments. Rule engine covers vowel sandhi (savarṇa-dīrgha, guṇa, vṛddhi, yān, ayādi), consonant sandhi, and visarga sandhi.

Transliteration

from sanskrit_tokenizer.transliterate import (
    devanagari_to_iast,
    iast_to_devanagari,
    is_devanagari,
)

devanagari_to_iast("संस्कृत")
# 'saṃskṛta'

iast_to_devanagari("dharma")
# 'धर्म'

is_devanagari("संस्कृत")
# True

Word-level tokenization

from sanskrit_tokenizer import tokenize_words

tokenize_words("dharma yoga namaḥ")
# ['dharma', 'yoga', 'namaḥ']

tokenize_words("धर्म योग")
# ['धर्म', 'योग']

tokenize_words() splits on whitespace and punctuation only — no sandhi splitting.

CLI

sanskrit-tokenize "dharma yoga"
# dharma
# yoga

sanskrit-tokenize "धर्म योग"
# धर्म
# योग

sanskrit-tokenize --no-sandhi "devālaya"
# devālaya

sanskrit-tokenize -s " " "dharma yoga"
# dharma yoga

sanskrit-tokenize --version
# sanskrit-tokenize 0.2.0

• --no-sandhi — word-level only, skip sandhi splitting
• --separator SEP — output separator (default: newline)
• --version — show version

License

MIT © Hemanth.HM