Process-Sanskrit

Process-Sanskrit is a python open-source library for automatic Sanskrit text annotation and inflected dictionary search.

The library has two main use cases:

1. Dictionary search: multi dictionary lookup with grammatical annotation for words exactly as they are written in texts: in any transliteration format, inflected, compounded and with sandhi.
2. Automatic Text Annotation: generate automatically version of Sanskrit texts without sandhi and with split compounds, with grammatical annotations and dictionary entries for each word.

The architecture of the library is based on a cascading approach to Sanskrit text analysis, as described in our NAACL 2025 paper: Accessible Sanskrit: A Cascading System for Text Analysis and Dictionary Access.

The library is one of the main components powering up the Sanskrit Voyager backend. It is used for dictionary searches, stemming, and to pre-process texts for the corpus search.

Demo:

The library can be employed live on the Sanskrit Voyager website.

Select a book or paste some text, click on the words and see the library in action!

Or search some inflected and sandhi-ed words in the search bar to get the dictionary entries.

The following is the Quickstart guide. For a more detailed documentation and advanced features refer to the documentation website.

Installation

To install the library use the standard pip install command, then call update-ps-database in the terminal (or process_sanskrit.update_database() from Python) to setup the database.

pip install process-sanskrit
update-ps-database

Everything the pipeline needs, including sandhi splitting with full statistical (DCS word2vec) scoring, is installed by the base package. There are no extras to remember.

Binary wheels for Linux (x86-64, aarch64), macOS (x86-64, arm64) and Windows (x86-64) ship the native Rust sandhi splitter, so installing needs no Rust or C++ toolchain. Building from source requires Rust 1.87 or newer and a C++17 compiler; see the Rust splitter guide for backend selection and build details.

update-ps-database downloads and setup the database with the dictionaries and the inflection tables (adjusted from CLS inflect ) in the resources folder (150 mb download, 583 mb uncompressed, released with Creative Commons NC license).

The same step is available from Python, which is handy in notebooks and scripts. It is idempotent, so calling it on every start-up costs nothing once the database is in place, and it raises process_sanskrit.DatabaseUpdateError instead of exiting on failure:

import process_sanskrit
process_sanskrit.update_database()   # returns the path of the ready database

Downloads are staged and atomically renamed into place, so an interrupted run never leaves a partial database behind. Details in documentation/database-setup.md.

## if inside jupyter or colab use:

!pip install process-sanskrit
!update-ps-database

only transliterate works without the database!

Deployments that keep the database outside the Python package can set PROCESS_SANSKRIT_DB_PATH to the existing SQliteDB.sqlite file. The library opens it read-only; see documentation/database-location.md.

Persistent analysis cache

Expensive sandhi and compound analyses are cached by default in a separate, disk-backed SQLite database. The dictionary database remains read-only. Cached results expire after 90 inactive days by default; set PROCESS_SANSKRIT_CACHE_RETENTION=keep_all to retain the de-duplicated prediction corpus indefinitely, or PROCESS_SANSKRIT_CACHE_ENABLED=false to disable persistence. Individual calls can pass cached=False or cached=True.

See the local cache guide for paths, privacy, retention, web-worker behaviour, and ML-corpus limitations.

Process Function:

The core of the library is the process function, that accepts text in Sanskrit as input and executes an entire text processing pipeline for the text.

import process-sanskrit as ps 

ps.process("pratiprasave")

Process returns a list that contains for each word contained in the text or compounds:

1. Word stem: ‘pratiprasava’
2. Grammatical tagging: masculine noun/adjective ending in a
3. Case (for nouns) or Inflection (for verbs): [('Loc', 'Sg')]
4. Inflection table for the word as a list: ['pratiprasavaḥ', 'pratiprasavau', 'pratiprasavāḥ', 'pratiprasavam', 'pratiprasavau', 'pratiprasavān', 'pratiprasavena', 'pratiprasavābhyām', 'pratiprasavaiḥ', 'pratiprasavāya', 'pratiprasavābhyām', 'pratiprasavebhyaḥ', 'pratiprasavāt', 'pratiprasavābhyām', 'pratiprasavebhyaḥ', 'pratiprasavasya', 'pratiprasavayoḥ', 'pratiprasavānām', 'pratiprasave', 'pratiprasavayoḥ', 'pratiprasaveṣu', 'pratiprasava', 'pratiprasavau', 'pratiprasavāḥ']
5. Original word: 'pratiprasave’
6. Word Components according to the Monnier Williams: (in this case none) 'prati—prasava’
7. Dictionary entries in XML format. In the form of a dictionary for all the selected dictionaries: {'mw': {'pratiprasava': ['prati—prasava a See under prati-pra- √ 1. sū.', 'prati-°prasava b m. counter-order, suspension of a general prohibition in a particular case, Śaṃkarācārya ; Kātyāyana-śrauta-sūtra , Scholiast or Commentator; Manvarthamuktāvalī, KullūkaBhaṭṭa's commentary on Manu-smṛti ', ' an exception to an exception, Taittirīya-prātiśākhya , Scholiast or Commentator', ' return to the original state, Yoga-sūtra ']}

Process automatically detects transliteration scheme and transliterate to IAST. If that is problematic, pre-transliterate to IAST first using the transliterate function.

Also, the base version of Process is optimised for single words, - if you have a sentence or book, split by spaces and pass each term to transliterate.

In the online interface it is possible to retrive the entries for the components (in this case 'prati' and 'prasava') by clicking on them. Clicking automatically sends to the dictionary entry of the components.

Dictionary Selection:

The process function returns dictionary entries for the found roots.

By default, only the Monnier Williams dictionary is selected.

In the following example we search for a word that is not in the MW: 'dvandva'. The process function automatically check if any of the dictionaries has it and automatically select it. In this case the word is found in the Macdonnell dictionary.

 
import process_sanskrit as ps
print(ps.process('dvandva'))

To use more dictionaries, process accepts as optional arguments the dictionary abbreviation. In the following code example we retrieve the entries for the word 'saṃskāra' from the Apte, Cappeller, Grassman, and Edgerton dictionaries.

import process_sanskrit as ps

print(ps.process('saṃskāra', 'ap90', 'cae', 'gra', 'bhs'))



### Available Dictionaries and Abbreviations 

Here is the list of all the currently available dictionaries with the abbreviations:

• 'mw': 'Monier-Williams Sanskrit-English Dictionary' ,
• 'ap90': 'Apte Practical Sanskrit-English Dictionary'
• ‘cae': 'Cappeller Sanskrit-English Dictionary'
• 'ddsa': 'Macdonell A Practical Sanskrit Dictionary'
• 'gra': 'Grassmann Wörterbuch zum Rig Veda'
• 'bhs': 'Edgerton Buddhist Hybrid Sanskrit Dictionary'
• 'cped': 'Concise Pali English Dictionary'

All the dictionaries are slightly modified version of the *Cologne Digital Sanskrit Dictionaries*, apart from the [The Concise Pali-English Dictionary By Buddhadatta Mahathera](https://buddhistuniversity.net/content/reference/concise-pali-dictionary). The Pali dictionary was added in to handle words that appears in the late Buddhist authors. 

Cologne Digital Sanskrit Dictionaries, version 2.7.286, Cologne University, accessed on February 19, 2025, https://www.sanskrit-lexicon.uni-koeln.de

### Stemming:

The process function can be used just for simple sandhi/compound split and stemming, adding the optional flag: *mode=’roots’*.

```python
import process_sanskrit as ps

print(ps.process('yamaniyamāsanaprāṇāyāmapratyāhāradhāraṇādhyānasamādhayo', mode='roots'))

## output:
## ['yama', 'niyama', 'asana', 'prāṇāyāma', 'pratyāhāra', 'dhāraṇa', 'dhyāna', 'samādhi']

In case of ambiguity the process function does not select between the two (or three) possibilities, but returns all of them.

Transliteration:

The library offers a function to transliterate texts with auto-detection for the transliteration input format. This function is a slight adaptation from Indic-Transliteration Detect.

import process_sanskrit as ps

# Transliteration
print(ps.transliterate("patañjali", "DEVANAGARI")) ## IAST 
print(ps.transliterate("pataJjali", "DEVANAGARI")) ## HK format

## same output:
## पतञ्जलि

## In case you need to manually select the input scheme, 
## force it using the optional 'input_scheme' flag
## the scheme it's not case sensitive (slp1=SLP1): 

print(ps.transliterate('pataYjali', 'tamil', input_scheme='slp1'))

## output: பதஞ்ஜலி

Dictionary Search:

The library provides the dict_search function to retrieve dictionary entries.

Pass to the dict_search a list of strings to be searched on and (optionally) a list of dictionary tags.

import process_sanskrit as ps

## unlike the process function, the dict_search wants the input in IAST format. 

# example usage for Dictionary lookup
ps.dict_search(['pratiprasava', 'saṃskāra'])

# after a list of entries, optionally add dictionary tags to search in multiple dictionaries. 

# search in Edgerton Buddhist Hybrid Sanskrit Dictionary
# and Grassmann Wörterbuch zum Rig Veda:
ps.dict_search(['pratiprasava', 'saṃskāra'], 'gra', 'bhs')

The library automatically handles the fact that the Apte records nominatives instead of un-inflected stems (i.E. yogaḥ instead of yoga).

ProcessBYT5

Experimental function: segments the text with the BYT5 model, then sends each word to the process function for stemming and grammatical results. The model needs torch and transformers, which the base package does not install, so this one feature needs the byt5 extra:

pip install process-sanskrit[byt5]

from process_sanskrit.functions.processBYT5 import processBYT5

processBYT5('śrutam āgamavijñānaṃ tat sāmānyaviṣayam')

Sources:

CLS inflect for the inflection tables: https://github.com/sanskrit-lexicon/csl-inflect

The Sanskrit Parser library by Karthik Madathil handles part of the Sandhi Splitting: https://github.com/kmadathil/sanskrit_parser

Since v1.5 a reduced, split-only copy of it is vendored under process_sanskrit/splitter/ (MIT licensed, and gratefully used). The sandhi splitting behaviour is unchanged — it is verified split-for-split against the original. See documentation/sandhi-splitter.md for what was kept, what was dropped, and why.

The split scorer uses a word2vec model trained on the Digital Corpus of Sanskrit (DCS), also from the Sanskrit Parser project: http://www.sanskrit-linguistics.org/dcs/

The BYT5 model used in the experimental version of the process function is from the https://huggingface.co/buddhist-nlp/byt5-sanskrit discussed in the paper:

One Model is All You Need: ByT5-Sanskrit, a Unified Model for Sanskrit NLP Tasks

Sebastian Nehrdich, Oliver Hellwig, Kurt Keutzer

https://arxiv.org/abs/2409.13920