sanskrit-tokenizer

Tokenize Sanskrit text with sandhi splitting for Information Retrieval.

pip install sanskrit-tokenizer

Quick start

from sanskrit_tokenizer import tokenize

tokenize("dharmakṣetre kurukṣetre")
# ['dharmakṣetre', 'kurukṣetre']

tokenize("धर्म योग")
# ['धर्म', 'योग']

tokenize("devālaya namaḥ")
# ['deva', 'ālaya', 'namaḥ']

tokenize() accepts both Devanagari and IAST input — output script matches input. Splits on whitespace and punctuation, applies reverse sandhi rules, and preserves the original script.

Sandhi splitting

from sanskrit_tokenizer.sandhi import split_sandhi

split_sandhi("devālaya")     # savarṇa-dīrgha: a + ā → ā
# ['deva', 'ālaya']

split_sandhi("rāmāyaṇa")    # savarṇa-dīrgha: a + ā → ā
# ['rāma', 'āyaṇa']

split_sandhi("देवालय")        # Devanagari in → Devanagari out
# ['देव', 'आलय']

split_sandhi("dharmakṣetre")  # no false-positive split
# ['dharmakṣetre']

Rule-based engine covering vowel sandhi (savarṇa-dīrgha, guṇa, vṛddhi, yān, ayādi), consonant sandhi (voicing, nasal assimilation), and visarga sandhi. Uses a scoring threshold to prevent false-positive splits.

Transliteration

from sanskrit_tokenizer.transliterate import (
    devanagari_to_iast,
    iast_to_devanagari,
    is_devanagari,
)

devanagari_to_iast("संस्कृत")
# 'saṃskṛta'

iast_to_devanagari("dharma")
# 'धर्म'

is_devanagari("संस्कृत")
# True

Word-level tokenization

from sanskrit_tokenizer import tokenize_words

tokenize_words("dharma yoga namaḥ")
# ['dharma', 'yoga', 'namaḥ']

tokenize_words("धर्म योग")
# ['धर्म', 'योग']

tokenize_words() splits on whitespace and punctuation only — no sandhi splitting.

CLI

sanskrit-tokenize "dharma yoga"
# dharma
# yoga

sanskrit-tokenize "धर्म योग"
# धर्म
# योग

sanskrit-tokenize --no-sandhi "devālaya"
# devālaya

sanskrit-tokenize -s " " "dharma yoga"
# dharma yoga

sanskrit-tokenize --version
# sanskrit-tokenize 0.2.0

• --no-sandhi — word-level only, skip sandhi splitting
• --separator SEP — output separator (default: newline)
• --version — show version

License

MIT © Hemanth.HM