process-sanskrit 1.0.10

A package automatic processing of Sanskrit texts and inflected dictionary search

Process-Sanskrit

Process-Sanskrit is a light-computing library for Sanskrit text analysis and annotation that requires no pre-processing.

The library has two main use cases:

1. Dictionary search: multi dictionary lookup with grammatical annotation for words exactly as they are written in texts: in any transliteration format, inflected, compounded and with sandhi.
2. Automatic Text Annotation: generate automatically version of Sanskrit texts without sandhi and with split compounds, with grammatical annotations and dictionary entries for each word.

The architecture of the library is based on a cascading approach to Sanskrit text analysis, as described in our NAACL 2025 paper:Accessible Sanskrit: A Cascading System for Text Analysis and Dictionary Access.

Demo:

The library can be employed live on the (Sanskrit Voyager website)[https://www.sanskritvoyager.com/].

Select a book or paste some text, click on the words and see the library in action!

Or search some inflected and sandhi-ed words in the search bar to get the dictionary entries.

The following is the Quickstart guide. For a more detailed documentation and advanced features refer to the documentation website.

Installation

To install the library use the standard pip command:

pip install process-sanskrit

After installing the library using pip, type update-ps-database in the command line.

The command downloads and setup the database with the dictionaries and the inflection tables in the resources folder (150 mb download, 583 mb uncompressed, (Creative Commons NC license)[ https://creativecommons.org/licenses/by-nc/4.0/]):

update-ps-database

-- if inside jupyter or colab use:

!update-ps-database

-- only the transliteration function works without the database! 

Process Function:

The core of the library is the process function, that accepts text in Sanskrit as input and executes an entire text processing pipeline for the text.

import process-sanskrit as ps 

ps.process("pratiprasave")

Process returns a list that contains for each word contained in the text or compounds:

1. Word stem: ‘pratiprasava’
2. Grammatical tagging: masculine noun/adjective ending in a
3. Case (for nouns) or Inflection (for verbs): [('Loc', 'Sg')]
4. Inflection table for the word as a list: ['pratiprasavaḥ', 'pratiprasavau', 'pratiprasavāḥ', 'pratiprasavam', 'pratiprasavau', 'pratiprasavān', 'pratiprasavena', 'pratiprasavābhyām', 'pratiprasavaiḥ', 'pratiprasavāya', 'pratiprasavābhyām', 'pratiprasavebhyaḥ', 'pratiprasavāt', 'pratiprasavābhyām', 'pratiprasavebhyaḥ', 'pratiprasavasya', 'pratiprasavayoḥ', 'pratiprasavānām', 'pratiprasave', 'pratiprasavayoḥ', 'pratiprasaveṣu', 'pratiprasava', 'pratiprasavau', 'pratiprasavāḥ']
5. Original word: 'pratiprasave’
6. Word Components according to the Monnier Williams: (in this case none) 'prati—prasava’
7. Dictionary entries in XML format. In the form of a dictionary for all the selected dictionaries: {'mw': {'pratiprasava': ['prati—prasava a See under prati-pra- √ 1. sū.', 'prati-°prasava b m. counter-order, suspension of a general prohibition in a particular case, Śaṃkarācārya ; Kātyāyana-śrauta-sūtra , Scholiast or Commentator; Manvarthamuktāvalī, KullūkaBhaṭṭa's commentary on Manu-smṛti ', ' an exception to an exception, Taittirīya-prātiśākhya , Scholiast or Commentator', ' return to the original state, Yoga-sūtra ']}

Dictionary Selection:

By default, only the Monnier Williams dictionary is selected.

If there are words that are outside the MW (i.E. ‘dvandva’) the dictionaries that have the word are automatically selected. In this example the word is found in the Macdonnell dictionary.

## additional dictionaries can be returned in the output by adding them as
## arguments after the main text input. 

import process-sanskrit as ps

ps.process('saṃskāra', 'ap90', 'cae', 'gra', 'bhs')

Available Dictionaries:

• 'mw': 'Monier-Williams Sanskrit-English Dictionary' ,
• 'ap90': 'Apte Practical Sanskrit-English Dictionary'
• ‘cae': 'Cappeller Sanskrit-English Dictionary'
• 'ddsa': 'Macdonell A Practical Sanskrit Dictionary'
• 'gra': 'Grassmann Wörterbuch zum Rig Veda'
• 'bhs': 'Edgerton Buddhist Hybrid Sanskrit Dictionary'
• 'cped': 'Concise Pali English Dictionary'

All the dictionaries are slightly modified version of the Cologne Digital Sanskrit Dictionaries, apart from the (The Concise Pali-English Dictionary By Buddhadatta Mahathera)[https://buddhistuniversity.net/content/reference/concise-pali-dictionary] The Pali dictionary was added in to handle words that appears in the late Buddhist authors.

Cologne Digital Sanskrit Dictionaries, version 2.7.286,
Cologne University, accessed on February 19, 2025,
https://www.sanskrit-lexicon.uni-koeln.de

Stemming:

To use the process function just for sandhi/compound split and stemming, use the process function with the flag: output=’roots’.

import process-sanskrit as ps

ps.process('yamaniyamāsanaprāṇāyāmapratyāhāradhāraṇādhyānasamādhayo', output='roots')

## output:
## ['yama', 'niyama', 'asana', 'prāṇāyāma', 'pratyāhāra', 'dhāraṇa', 'dhyāna', 'samādhi']

!it is to note that in case of ambiguity the process function does not select between the two (or three) possibilities, but returns all of them as a tuple.

Transliteration API:

The library offers a function to transliterate texts with auto-detection for the transliteration input format. This function is a slight adaptation from (Indic-Transliteration Detect)[”https://github.com/indic-transliteration/detect.py”].

import process_sanskrit as ps

# Transliteration
ps.transliterate("patañjali", "DEVANAGARI") ## IAST 
ps.transliterate("pataJjali", "DEVANAGARI") ## HK format

## same output:
## पतञ्जलि

## In case you need to manually select the input scheme, you can force it using
## the input_scheme flag, it's not case sensitive: 

ps.transliterate('pataYjali', 'tamil', input_scheme='slp1')

## output: பதஞ்ஜலி

Dictionary Search:

The library provides the dict_search function to retrieve dictionary entries.

Pass to the dict_search a list of strings to be searched on and (optionally) a list of dictionary tags.

import process_sanskrit as ps

## unlike the process function, the dict_search wants the input in IAST format. 

# example usage for Dictionary lookup
ps.dict_search(['pratiprasava', 'saṃskāra'])

# after a list of entries, optionally add dictionary tags to search in multiple dictionaries. 

# search in Edgerton Buddhist Hybrid Sanskrit Dictionary
# and Grassmann Wörterbuch zum Rig Veda:
ps.dict_search(['pratiprasava', 'saṃskāra'], 'gra', 'bhs')

The library automatically handles the fact that the Apte records nominatives instead of un-inflected stems (i.E. yogaḥ instead of yoga).

Sources:

CLS inflect for the inflection tables: https://github.com/sanskrit-lexicon/csl-inflect

The Sanskrit Parser library handles part of the Sandhi Splitting: https://github.com/kmadathil/sanskrit_parser?tab=readme-ov-file

The BYT5 model used in the experimental version of the process function is from the https://huggingface.co/buddhist-nlp/byt5-sanskrit discussed in the paper:

One Model is All You Need: ByT5-Sanskrit, a Unified Model for Sanskrit NLP Tasks

Sebastian Nehrdich, Oliver Hellwig, Kurt Keutzer

https://arxiv.org/abs/2409.13920

The current implementation requires about 2gb of ram in total without a GPU. The version of process that employs BYT5 for sandhi removal and compound splitting (stemming is still handled by the inflection tables) instead of Sanskrit Parser requires about 6gb including RAM and VRAM.