sandesh-unicode 1.0.1

1-byte encoding for Nepali text - 66% smaller than UTF-8

सन्देश Unicode

Efficient 1-Byte Encoding for Nepali Text — सन्देश encodes Nepali Unicode characters as single bytes instead of the 3-byte UTF-8 representation, achieving ~66% size reduction for pure Nepali text.

Metric	UTF-8	सन्देश	Savings
Pure Nepali	3 bytes/ch	1 byte/ch	~66%
Mixed Nepali+English	varies	varies	~40-50%
ASCII only	1 byte/ch	1 byte/ch	0%

---

Installation

pip install sandesh-unicode

For development:

pip install "sandesh-unicode[dev]"

---

Quick Start

CLI

# Encode Nepali text
sandesh encode "सन्देश"
# Output: b'\xaa\x9e\x9c\xb2\xa8'

# Decode bytes back to text
echo -ne '\xaa\x9e\x9c\xb2\xa8' | sandesh decode
# Output: सन्देश

# Show compression statistics
sandesh stats "सन्देश नेपाल"

# Encode/decode files
sandesh encode-file input.txt output.sndsh
sandesh decode-file output.sndsh decoded.txt

# View mapping table
sandesh table

# Package info
sandesh info

Python API

from sandesh_unicode import encode, decode, get_compression_ratio

# Basic encode/decode
encoded = encode("सन्देश")
print(encoded)           # b'\xaa\x9e\x9c\xb2\xa8'
decoded = decode(encoded)
print(decoded)           # सन्देश

# Mixed Nepali-English text
encoded = encode("नमस्ते नेपाल")
decoded = decode(encoded)
print(decoded)           # नमस्ते नेपाल

# Compression statistics
stats = get_compression_ratio("सन्देश")
print(f"Saved {stats['savings_percent']:.1f}%")  # Saved ~66.7%

# Permissive mode (skip unsupported characters)
encoded = encode("Hello सन्देश 中文", strict=False)

# File operations
from sandesh_unicode import encode_file, decode_file
encode_file("nepali.txt", "nepali.sndsh")
decode_file("nepali.sndsh", "nepali_decoded.txt")

# Validation
from sandesh_unicode import is_sandesh_encoded, validate
data = encode("सन्देश")
print(is_sandesh_encoded(data))   # True
errors = validate(data)
print(len(errors))                # 0

---

Architecture

┌─────────────┐     encode()     ┌──────────────┐
│  Unicode     │ ──────────────→ │  1-byte code  │
│  (UTF-8)     │ ←────────────── │  (0x80-0xFF)  │
│  3 bytes/ch  │     decode()    │  1 byte/ch    │
└─────────────┘                  └──────────────┘
     │                                │
     │ UTF-8 preserved for ASCII      │ ASCII passthrough
     │ (0x00-0x7F)                    │ (0x00-0x7F)
     └────────────────────────────────┘

Memory Layout

Range	Category	Count
0x00-0x7F	ASCII (passthrough)	128
0x80-0x8A	Vowels (अ-औ)	11
0x8B-0xAB	Consonants (क-ह)	33
0xAC-0xB6	Matras/Vowel signs	11
0xB7-0xBA	Special signs	4
0xBB-0xBC	Punctuation	2
0xBE-0xC7	Nepali digits (०-९)	10
0xC8-0xFF	Common conjuncts	56

---

Use Cases

• Database storage: Store Nepali text using 66% less space
• File compression: Compress .txt files containing Nepali text
• Network transfer: Reduce bandwidth for Nepali text payloads
• Embedded systems: Efficient storage on memory-constrained devices
• APIs: Optimize Nepali text in REST/GraphQL payloads

See the examples/ directory for backend (FastAPI) and frontend (JavaScript) implementations.

---

Development

# Clone and install with dev dependencies
git clone https://github.com/sandesh-unicode/sandesh-unicode.git
cd sandesh-unicode
pip install -e ".[dev]"

# Run tests
pytest

# Run tests with coverage
pytest --cov=sandesh_unicode --cov-report=term

# Format code
black src/ tests/
isort --profile black src/ tests/

# Run pre-commit hooks
pre-commit run --all-files

Project Structure

sandesh-unicode/
├── src/
│   └── sandesh_unicode/
│       ├── __init__.py      # Public API exports
│       ├── tables.py         # Character mapping tables
│       ├── encoder.py        # Encode Unicode → सन्देश bytes
│       ├── decoder.py        # Decode सन्देश bytes → Unicode
│       ├── exceptions.py     # Custom exception hierarchy
│       └── cli.py            # Command-line interface
├── tests/
│   └── test_sandesh.py       # Comprehensive test suite
├── examples/
│   ├── backend/              # FastAPI storage API example
│   └── frontend/             # Browser-based encoder/decoder
├── pyproject.toml            # Build & tool configuration
└── .github/workflows/ci.yml  # CI pipeline (3 OS × 6 Python versions)

---

Limitations

• Ecosystem-dependent: Only works within systems using this library
• Display-agnostic: Must decode to Unicode for standard display
• Not a Unicode replacement: Internal compression scheme only
• Nepali-only: Currently supports Devanagari Nepali script

---

License

MIT