tokon 1.1.0

Token-Optimized Serialization Format for AI-Native Applications

Tokon v1.1

Token-Optimized Serialization Format for AI-Native Applications

Tokon is a dual-mode, schema-driven data serialization format designed for maximum token efficiency and human readability.

Inspired by the TOON format, but completely redesigned for AI-native workflows with dual-mode architecture and schema-driven optimization.

Features

• 78% token reduction vs JSON in compact mode
• 53% token reduction vs JSON in human mode
• Dual representation - readable for humans, compact for LLMs
• Schema-driven - stable symbols across projects
• Type-safe - built-in validation
• Streaming-ready - incremental parsing support
• Zero dependencies - pure Python

Installation

pip install tokon

Quick Start

from tokon import encode, decode

# Encode to human-readable format
data = {"name": "Alice", "age": 30, "active": True}
tokon_h = encode(data, mode='h')
print(tokon_h)
# name Alice
# age 30
# active true

# Decode back
decoded = decode(tokon_h, mode='h')

See QUICK_START.md for more examples.

Modes

Tokon-H (Human Mode)

Clean, readable format:

user
  name Alice
  age 30
  active true

Tokon-C (Compact Mode)

Ultra-compact with schemas:

u[n:Alice a:30 x:1]

Documentation

• QUICK_START.md - Get started in 5 minutes
• INSTALLATION.md - Installation guide
• USAGE.md - Complete usage guide
• TOKON_SPEC.md - Full specification
• KNOWN_LIMITATIONS.md - Edge cases

Command Line

# Encode JSON to Tokon
echo '{"name": "Alice"}' | tokon encode -m h

# Decode Tokon to JSON
echo 'name Alice' | tokon decode

Performance

• Token Efficiency: 53-78% reduction vs JSON
• Speed: Comparable to JSON
• Memory: Efficient for large datasets

Requirements

• Python 3.8+
• No dependencies

License

MIT License - see LICENSE file for details.

Version

Tokon v1.1.0 - Production Ready