santok 1.0.6

Advanced multi-format tokenization system with numerology, hashing, compression, and embeddings

SanTOK - Advanced Multi-Format Tokenization System

SanTOK (Sanitized Tokenization) is an advanced multi-format tokenization system that provides 9 different tokenization methods with integrated numerology, hashing, compression, and embedding capabilities.

🚀 Features

Core Tokenization Methods

• Space Tokenization: Splits text on whitespace
• Word Tokenization: Extracts words using regex patterns
• Character Tokenization: Character-by-character analysis
• Grammar Tokenization: Separates words, numbers, and punctuation
• Subword Tokenization: Fixed-length chunking
• Byte Tokenization: ASCII value representation
• BPE Tokenization: Byte Pair Encoding
• Syllable Tokenization: Vowel-based splitting
• Frequency Tokenization: Word frequency analysis

Advanced Features

• Numerology Integration: 9-centric digital root calculations
• Hash-Driven Embeddings: Stable across vocabularies
• Lossless Reconstruction: Perfect text reconstruction
• Multi-Format Output: JSON, CSV, TXT, XML, Excel, Parquet, Avro
• High Performance: Concurrent and async processing

📦 Installation

pip install santok

🎯 Quick Start

import santok

# Basic usage
text = "Hello world!"
result = santok.all_tokenizations(text)

# Access different tokenization methods
space_tokens = result['space']
char_tokens = result['char']
word_tokens = result['word']

print(f"Space tokens: {space_tokens}")
print(f"Character tokens: {char_tokens}")
print(f"Word tokens: {word_tokens}")

# Numerology calculation
numerology = santok.numerology_sum(text)
print(f"Numerology sum: {numerology}")

📊 Output Format

Each tokenization method returns a list of dictionaries:

[
    {'text': 'Hello', 'frontend': 1},
    {'text': 'world!', 'frontend': 2}
]

Where:

• text: The actual token
• frontend: Numerological frontend digit (1-9)

🔄 Lossless Reconstruction System

Lossless Reconstruction Methods (3/9 methods)

• ✅ SPACE: Preserves all whitespace and punctuation perfectly
• ✅ CHAR: Character-by-character perfect preservation
• ✅ BPE: Advanced subword with full structure preservation

Analytical Methods (6/9 methods - Transform text for analysis)

• 🔄 WORD: Extracts words for linguistic analysis (removes punctuation by design)
• 🔄 GRAMMAR: Parses grammatical elements (removes spacing by design)
• 🔄 SUBWORD: Fixed-length chunking for subword modeling (transforms by design)
• 🔄 BYTE: ASCII representation for byte-level analysis (different format by design)
• 🔄 SYLLABLE: Syllable extraction for phonetic analysis (removes spacing by design)
• 🔄 FREQUENCY: Adds frequency metadata for statistical analysis (enhances by design)

🛠️ Advanced Usage

CLI Usage

santok

Programmatic Usage

import santok

# Get all tokenizations
result = santok.all_tokenizations("Your text here")

# Calculate numerology
numerology = santok.numerology_sum("Your text here")

# Run main function
santok.main()

📈 Performance

• Concurrent Processing: Multi-threaded tokenization
• Async Support: Asynchronous processing for large texts
• Memory Efficient: Stream processing for large datasets
• High Speed: Optimized algorithms for maximum performance

🔧 Requirements

• Python 3.8+
• No external dependencies (pure Python)

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

👨‍💻 Author

Santosh chavala

• Email: [e-mail]
• GitHub: @chavalasantosh

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

📝 Changelog

See CHANGELOG.md for a list of changes and version history.

🔗 Links

• PyPI Package
• GitHub Repository
• Documentation
• Issue Tracker

---

SanTOK - Advanced Multi-Format Tokenization System by Santosh chavala