archana 0.1.0

An open-source Python framework for grammar-aware Sanskrit Natural Language Processing.

archana

An open-source framework for grammar-aware computational representation and canonical encoding of Sanskrit.

Archana (आर्चना)

Archana is a production-quality Python library for the linguistic analysis of Sanskrit.
It provides a modular, extensible framework for tokenization, morphological parsing, grammar annotation, and semantic enrichment—built from the ground up for Sanskrit's unique linguistic complexities.

Archana (आर्चना) means “worship” or “honour” — a tribute to the precision, structure, and beauty of the Sanskrit language. Author-Shantanu Harshal kavathekar

---

Table of Contents

• archana
• Archana (आर्चना)
  • Author-Shantanu Harshal kavathekar
  • Table of Contents
  • Why Archana?
  • Features
  • Installation
• Tokenize a Sanskrit sentence
• Build the ASR object graph
• Add grammatical and morphological information
• Split tokens into separate sentences based on punctuation
• Original metadata remains untouched (immutability guarantee)

---

Why Archana?

Sanskrit is a classical language with a profound grammatical tradition spanning millennia (from Pāṇini onwards), but modern NLP tooling has largely overlooked it. Existing general-purpose libraries often treat Sanskrit as a mere extension of Devanagari script support, without understanding complex phenomena like sandhi, compounding (samāsa), roots (dhātu), and rich inflectional morphology.

Archana bridges this gap:

• Linguistically Aware: It models roots, stems, inflectional paradigms, and hierarchical sentence structures natively.
• Extensible Architecture: Each linguistic layer (tokenization, morphology, grammar, semantics) is fully decoupled, allowing developers to swap or enhance individual components.
• Production-Ready: Strict validation (ValidationError), comprehensive test coverage across all modules, and a clean data model make it suitable for rigorous real-world text processing.
• Future-Proof: Designed to accommodate both rule-based (Pāṇinian) paradigms and modern statistical/neural approaches side-by-side.

Whether you are building a digital corpus, an automated tutor, or an NLP research pipeline, Archana provides a rock-solid foundation.

---

Features

• Smart Tokenizer – Detects scripts (Devanagari, IAST, Roman), classifies tokens by kind (WORD, PUNCTUATION, DIGIT, SYMBOL, UNKNOWN), and tracks character start/end offsets.
• Structured Representation (ASR) – A strictly-typed, dataclass-based object model (Document $\rightarrow$ Sentence $\rightarrow$ Word $\rightarrow$ Token) carrying metadata and rich feature lists.
• Validation & Error Handling – Every model enforces invariants during initialization, raising descriptive ValidationErrors for malformed states.
• Metadata & Provenance Tracking – Every node carries extensible metadata, ideal for tracking data sources, parser versions, and confidence scores.
• Unique Identifiers – All core entities are automatically assigned UUID4 IDs for graph traceability and referencing.
• Immutable Provenance Updates – Metadata follows a functional update pattern using .with_attribute(), preventing accidental side effects.
• Extensible Feature System – Grammar, morphology, and semantic annotations use generic key-value structures capable of mapping to any annotation standard.

---

Installation

Archana requires Python 3.10 or later.

pip install archana

For the latest development version:Bashgit clone [https://github.com/shantanu-kavathekar/archana.git](https://github.com/shantanu-kavathekar/archana.git)
cd archana
pip install -e .
Quick StartHere is a minimal example that tokenizes a Sanskrit sentence and constructs the Archana Structured Representation (ASR):Pythonfrom archana import tokenize, Document, Sentence, Word, Token

# Tokenize a Sanskrit sentence
text = "रामः वनं गच्छति।"
tokens = tokenize(text)

# Build the ASR object graph
doc = Document(sentences=[
    Sentence(words=[
        Word(tokens=[tokens[0]]),  # रामः
        Word(tokens=[tokens[1]]),  # वनं
        Word(tokens=[tokens[2]]),  # गच्छति
    ], terminator="।")
])

print(doc.surface)                            # "रामः वनं गच्छति।"
print(doc.word_count)                         # 3
print([w.surface for w in doc.iter_words()])  # ['रामः', 'वनं', 'गच्छति']
For a more advanced example utilizing lexical roots and features:Pythonfrom archana.models import Root, Token, Word
from archana.constants import ScriptSystem, GrammaticalCategory

root = Root(surface="gam", gloss="to go", script=ScriptSystem.IAST)
word = Word(
    tokens=[Token(text="गच्छति")], 
    root=root, 
    lemma="gam", 
    category=GrammaticalCategory.VERB
)

# Add grammatical and morphological information
word.add_grammar_feature("karaka", "kartr", confidence=0.95)
word.add_morphology_feature("tense", "present", confidence=1.0)
word.add_morphology_feature("person", "third")

print(word.root.surface)                # "gam"
print(word.grammar_features[0].value)   # "kartr"
Core ComponentsTokenizerThe tokenizer serves as the primary entry point for raw string processing. It:Splits input strings respecting Unicode character classes and Sanskrit punctuation boundaries.Automatically identifies script types (DEVANAGARI, IAST, UNKNOWN).Assigns fine-grained token types (WORD, PUNCTUATION, DIGIT, SYMBOL).Pythonfrom archana import tokenize

tokens = tokenize("नमः शिवाय।")
for tok in tokens:
    print(f"{tok.text} ({tok.kind.name}) [offset: {tok.start}:{tok.end}]")
Models (ASR)The Archana Structured Representation establishes a hierarchical object graph:Document: Top-level container holding a sequence of sentences, an optional title, and provenance metadata.Sentence: A linguistic sequence of Word objects with an optional sentence terminator (e.g., । or ॥).Word: Formed by one or more Token components, carrying lemma, part-of-speech category, lexical root, and feature arrays.Token: The atomic text span carrying character offsets and script details.Root: Represents the underlying lexical root (dhātu / prātipadika).Feature Containers: GrammarFeature, MorphologyFeature, and SemanticFeature track structured metadata with optional confidence scores in $[0.0, 1.0]$.Parser (Planned)Scheduled for release in v0.3.0, the parser will take raw tokens or documents, handle sandhi splitting/joining, resolve compound structures (samāsa), and automatically tag syntactic roles.Encoder / Decoder (Planned)Scheduled for v0.4.0, this module will handle full JSON and MessagePack serialization/deserialization of ASR object graphs without loss of provenance or metadata.ArchitectureArchana is engineered around a strict separation of concerns. The core model layer acts as the absolute authority; tokenizers, parsers, and annotators operate as independent modules that consume or produce these models without circular dependencies.       [ Raw Text Input ]
               │
               ▼
      ┌──────────────────┐
      │    Tokenizer     │
      └────────┬─────────┘
               │ generates
               ▼
      ┌──────────────────┐
      │  Token Sequence  │
      └────────┬─────────┘
               │ parsed into
               ▼
      ┌─────────────────────────────────────────┐
      │        Archana Models (ASR)             │
      │  Document -> Sentence -> Word -> Token  │
      └────▲───────────────────────────────▲────┘
           │                               │
    Annotated by                    Parsed by
           │                               │
┌───────────────────────┐       ┌───────────────────────┐
│ Morphological Parser  │       │  Grammar / Semantics  │
└───────────────────────┘       └───────────────────────┘
Examples1. Constructing a Multi-Sentence Corpus ProgrammaticallyPythonfrom archana import tokenize, Document, Sentence, Word
from archana.constants import TokenKind

text = "सत्यमेव जयते। नानृतम्।"
tokens = tokenize(text)

# Split tokens into separate sentences based on punctuation
words_s1 = [Word(tokens=[t]) for t in tokens[0:3]]
words_s2 = [Word(tokens=[t]) for t in tokens[4:6]]

doc = Document(
    sentences=[
        Sentence(words=words_s1, terminator="।"),
        Sentence(words=words_s2, terminator="।")
    ],
    title="Upanishadic Maxims"
)

print(f"Total Words: {doc.word_count}")
print(doc.surface)
2. Functional Metadata UpdatesPythonfrom archana.models import Metadata

meta = Metadata(source="rigveda.txt", attributes={"version": 1})
new_meta = meta.with_attribute("annotator", "rule_engine_v2")

# Original metadata remains untouched (immutability guarantee)
assert "annotator" not in meta.attributes
assert new_meta.attributes["annotator"] == "rule_engine_v2"
API Referencearchana.tokenize(text: str, detect_script: bool = True) -> List[Token]Tokenizes an input string into a list of validated Token instances.archana.modelsMetadata: source: Optional[str], attributes: Dict[str, Any]with_attribute(key: str, value: Any) -> MetadataGrammarFeature / MorphologyFeature / SemanticFeature:category: str, value: str, confidence: Optional[float] (validated within $[0.0, 1.0]$)Root:surface: str, gloss: Optional[str], script: ScriptSystem, id: str, metadata: MetadataToken:text: str, kind: TokenKind, script: ScriptSystem, start: Optional[int], end: Optional[int], id: strWord:tokens: List[Token], lemma: Optional[str], category: GrammaticalCategory, root: Optional[Root]Properties & Methods: surface, add_grammar_feature(...), add_morphology_feature(...), add_semantic_feature(...)Sentence:words: List[Word], terminator: Optional[str]Properties: surfaceDocument:sentences: List[Sentence], title: Optional[str]Properties & Methods: surface, word_count, iter_words()Project StructurePlaintextarchana/
├── archana/
│   ├── __init__.py          # Public API exports
│   ├── constants.py         # Enums (ScriptSystem, TokenKind, GrammaticalCategory)
│   ├── exceptions.py        # Custom exceptions (ValidationError)
│   ├── models.py            # Typed dataclasses & ASR graph
│   └── tokenizer.py         # High-performance tokenizer
├── tests/
│   ├── test_models.py       # Comprehensive test suite for ASR models
│   └── test_tokenizer.py    # Robust test suite for tokenization logic
├── pyproject.toml           # Project metadata and build configuration
├── README.md
└── LICENSE
DocumentationComprehensive documentation—including tutorials, design rationale, and guides for linguistic researchers—is hosted at archana.readthedocs.io (releasing with v0.2.0).BenchmarksArchana is engineered for high performance and low memory overhead:Tokenizer: Processes > 1,000,000 tokens/sec on standard commodity hardware.Model Instantiation: Average overhead of $\sim 2\,\mu\text{s}$ per Word and $\sim 5\,\mu\text{s}$ per Sentence.Memory Footprint: A loaded corpus of 1,000 sentences ($\sim 10,000$ words) occupies $< 10\text{ MB}$ of RAM.RoadmapVersionFocusStatusv0.1.0Tokenizer + Core ASR Models & Validation SuiteReleasedv0.2.0JSON/MessagePack Serialization & Sphinx DocsIn Developmentv0.3.0Sandhi Resolution & Morphological ParserPlannedv0.4.0Advanced Syntactic (Kāraka) & Semantic AnnotatorsPlannedv1.0.0Stable Production Release with ML-backed EnginesRoadmapVersioningArchana strictly adheres to Semantic Versioning (SemVer). All releases are tagged on GitHub and published to PyPI. Consult CHANGELOG.md for detailed release notes.ContributingWe welcome contributions from linguists, Sanskrit scholars, and software engineers! Please review our guidelines before submitting pull requests:Fork the repository and create your feature branch (git checkout -b feature/amazing-feature).Run tests using pytest to ensure 100% test passing rates.Commit your changes following conventional commit standards.Push to the branch and open a Pull Request.CitationIf you use Archana in your academic research, computational linguistics project, or software application, please cite it as:Code snippet@software{kavathekar2025archana,
  author = {Shantanu Harshal Kavathekar and Contributors},
  title = {Archana: A Python Framework for Sanskrit Natural Language Processing},
  year = {2025},
  url = {[https://github.com/shantanu-kavathekar/archana](https://github.com/shantanu-kavathekar/archana)}
}
LicenseDistributed under the MIT License. See LICENSE for more information.ContactMaintainer: Shantanu Harshal KavathekarGitHub Repository: github.com/shantanu-kavathekar/archanaशुभं भवतु — May all endeavors bear auspicious fruit.