toonstream 2.0.0

Token Oriented Object Notation - Efficient data serialization for LLMs with TOON and TRON formats

🎨 ToonStream

Token-Oriented Object Notation (TOON) & Token Reduced Object Notation (TRON) - Reduce LLM token usage by up to 73% with lossless data serialization

---

📖 What is ToonStream?

ToonStream is a Python library for encoding structured data in a token-efficient format designed for Large Language Models (LLMs). It converts repetitive JSON structures into compact, tabular representations that dramatically reduce token count while maintaining 100% lossless conversion.

The Problem

LLMs charge by tokens. Verbose JSON wastes tokens and money:

[
  {"id": 1, "name": "Alice", "dept": "Engineering", "salary": 95000},
  {"id": 2, "name": "Bob", "dept": "Sales", "salary": 75000},
  {"id": 3, "name": "Carol", "dept": "Engineering", "salary": 105000}
]

Cost: 80 tokens

The Solution

TOON format eliminates redundancy:

employees[3]{id,name,dept,salary}:
1,Alice,Engineering,95000
2,Bob,Sales,75000
3,Carol,Engineering,105000

Cost: 38 tokens (-52.5% reduction)

TRON format (NEW in v2.0.0) - ultra-compact:

@id,name,dept,salary|1,Alice,Engineering,95000|2,Bob,Sales,75000|3,Carol,Engineering,105000

Cost: 28 tokens (-65% reduction)

Why ToonStream?

✅ Save Money - Reduce API costs by up to 73% on structured data
✅ Two Formats - TOON (tabular) and TRON (ultra-compact)
✅ 100% Lossless - Perfect round-trip conversion, no data loss
✅ Zero Dependencies - Pure Python, no external packages required
✅ Fast - Sub-millisecond encoding/decoding
✅ Smart - Automatic optimization, only improves when beneficial
✅ Simple API - encode(data, format='tron') and decode(data, format='tron')

---

🚀 Installation

pip install toonstream

Or from source:

git clone https://github.com/vivekpandian08/toonstream.git
cd toonstream
pip install -e .

Requirements:

• Python 3.8 or higher
• No external dependencies (tiktoken optional for benchmarks)

Basic Usage

import toonstream

# Your data
data = {
    "name": "Alice",
    "age": 30,
    "skills": ["Python", "JavaScript", "SQL"]
}

# Encode to TOON
toon_str = toonstream.encode(data)
print(toon_str)

Output:

name: "Alice"
age: 30
skills: [
  - "Python"
  - "JavaScript"
  - "SQL"
]

🖥️ CLI Tool

ToonStream includes a command-line interface for easy file conversion.

# Convert JSON to TOON (default)
toonstream encode input.json -o output.toon

# Convert JSON to TRON
toonstream encode input.json --format tron -o output.tron

# Decode TOON/TRON back to JSON
toonstream decode output.toon -o restored.json

---

⚡ Quick Start

TOON Format (Tabular)

from toonstream import encode, decode

# Your data
data = {
    "employees": [
        {"id": 1, "name": "Alice", "dept": "Engineering"},
        {"id": 2, "name": "Bob", "dept": "Sales"},
        {"id": 3, "name": "Carol", "dept": "Engineering"}
    ]
}

# Encode to TOON format (default)
toon_str = encode(data)
print(toon_str)
# Output:
# employees[3]{id,name,dept}:
# 1,Alice,Engineering
# 2,Bob,Sales
# 3,Carol,Engineering

# Decode back to Python
decoded = decode(toon_str)
assert decoded == data  # ✓ Perfect round-trip!

TRON Format (Ultra-Compact) - NEW in v2.0.0

from toonstream import encode, decode

# Simple object
data = {"name": "Alice", "age": 30, "active": True}

# Encode to TRON format
tron_str = encode(data, format='tron')
print(tron_str)
# Output: name=Alice;age=30;active=1

# Nested objects use dot notation
nested = {"user": {"profile": {"name": "Bob", "city": "NYC"}}}
print(encode(nested, format='tron'))
# Output: user.profile.name=Bob;user.profile.city=NYC

# Tabular data uses @header|row format
employees = [
    {"id": 1, "name": "Alice", "dept": "Engineering"},
    {"id": 2, "name": "Bob", "dept": "Sales"},
]
print(encode(employees, format='tron'))
# Output: @id,name,dept|1,Alice,Engineering|2,Bob,Sales

# Decode TRON back to Python
decoded = decode(tron_str, format='tron')
assert decoded == data  # ✓ Perfect round-trip!

Smart Mode Selection with auto_mode

New in v1.1.0: Single parameter for intelligent mode detection

# Auto mode - automatically detects tensor data
toon_str = encode(data, auto_mode=True)
decoded = decode(toon_str, auto_mode=True)

# With PyTorch tensors (auto_mode detects and preserves them)
import torch
data_with_tensors = {
    'embeddings': torch.randn(10, 768),
    'labels': [0, 1, 0],
    'metadata': {'model': 'bert-base'}
}

# auto_mode automatically handles tensor serialization
encoded = encode(data_with_tensors, auto_mode=True)
decoded = decode(encoded, auto_mode=True)
# ✓ Tensors preserved with metadata (dtype, device, shape)

Advanced Options

# Compact mode (minimize whitespace)
compact = encode(data, compact=True)

# Disable smart optimization (always use tabular)
always_tabular = encode(data, smart_optimize=False)

# Pretty print with indentation
pretty = encode(data, indent=2)

# Sort dictionary keys
sorted_output = encode(data, sort_keys=True)

# Combine with auto_mode
combined = encode(data, auto_mode=True, compact=True)

---

🤔 Which Format Should I Use?

Feature	JSON	TOON	TRON
Human Readable	⭐⭐⭐	⭐⭐	⭐
Token Efficiency	⭐	⭐⭐	⭐⭐⭐
Parsing Speed	⭐⭐⭐	⭐⭐⭐	⭐⭐⭐
Typical Savings	0%	40-55%	50-70%

📋 JSON

Use when:

• Human readability is the #1 priority.
• Debugging raw payloads manually.
• Interoperating with systems that strictly require JSON.

📊 TOON (Tabular)

Use when:

• You have lists of objects (e.g., database rows, logs, CSV-like data).
• You want significant compression but still want to be able to read the data reasonably well.
• Example: User lists, product catalogs, transaction histories.

🚀 TRON (Compact)

Use when:

• Token cost or Context Window space is critical.
• Sending large contexts to LLMs (RAG, long histories).
• Data has deep nesting or mixed types.
• Example: Complex configs, API responses, arbitrary object trees.

---

📊 Performance Benchmarks

Real-world results using tiktoken (GPT-3.5/GPT-4 tokenizer):

Format Comparison (45 Complex Examples)

Format	Total Tokens	vs JSON	Best For
JSON (pretty)	16,583	--	Human readability
JSON Compact	9,713	-41%	Network transmission
TOON	7,144	-57%	Large tabular data
TRON	6,431	-61%	Most use cases

Detailed Results by Data Type

Data Type	JSON	Compact	TOON	TRON	TRON Savings
Simple Object	41	26	28	24	+41.5%
Employee Records (5)	168	88	56	52	+69.0%
User Records (20)	842	482	252	288	+65.8%
Nested Object	58	29	38	27	+53.4%
Mixed Structure	81	40	33	28	+65.4%
Products (50)	2,116	1,216	677	723	+65.8%

When to Use Each Format

Use Case	Best Format	Token Savings
Simple key-value objects	TRON	40%+
Nested configurations	TRON	50%+
Large tabular data (50+ rows)	TOON	45%+
Mixed nested + arrays	TRON	65%+
Streaming to LLMs	TRON	60%+

🟢 TRON wins 87% of benchmarks - use it for most LLM applications!

• Small datasets (<3 items)

Speed

All operations complete in under 1 millisecond for typical datasets:

• 50 records: 0.41ms
• 100 records: 0.83ms
• Decode: <1ms

---

🎯 Use Cases

1. LLM Context Optimization

3. Install in development mode:

pip install -e .

4. Install development dependencies (optional):

pip install -e ".[dev]"

This includes:

• pytest - Testing framework
• pytest-cov - Coverage reporting
• tiktoken - Token counting
• black - Code formatting

Verify Installation

# Run tests
pytest tests/test_toonstream.py

# Run benchmarks
python benchmarks/run_all_comparisons.py

# Try the tutorial
jupyter notebook examples/toonstream_tutorial.ipynb

Project Structure

toonstream/
├── toonstream/           # Core library
│   ├── __init__.py       # Public API exports
│   ├── encoder.py        # TOON encoder
│   ├── decoder.py        # TOON decoder
│   ├── tron_encoder.py   # TRON encoder (NEW in v2.0.0)
│   ├── tron_decoder.py   # TRON decoder (NEW in v2.0.0)
│   ├── unified_api.py    # Unified encode/decode API
│   ├── cli.py            # CLI implementation (NEW v2.0.0)
│   ├── exceptions.py     # Exception hierarchy
│   └── pickle_utils.py   # Pickle integration
├── benchmarks/           # Performance tests
├── tests/                # Test suite (186 tests, 100% passing)
├── examples/             # Usage examples
├── data/                 # Benchmark datasets
├── results/              # Benchmark results
├── README.md             # This file
├── PICKLE_USAGE.md       # Pickle utilities guide
├── pyproject.toml        # Modern package configuration
├── .pre-commit-config.yaml # Code quality hooks
├── setup.py              # Package configuration
└── requirements.txt      # Dependencies

1. LLM Context Optimization

import toonstream

# Pass structured data to LLM
context = {
    "users": [...],  # 100 user records
    "products": [...],  # 50 products
    "orders": [...]  # 200 orders
}

# Reduce prompt tokens by 40%
toon_context = toonstream.encode(context)
response = llm.complete(f"Analyze this data:\n{toon_context}")

2. Pickle Integration

Save data with TOON encoding for additional compression:

from toonstream import save_toon_pickle, load_toon_pickle

# Save with TOON encoding
data = {"users": [...], "logs": [...]}
save_toon_pickle(data, 'data.toon.pkl')

# Load back
loaded = load_toon_pickle('data.toon.pkl')

# 11.4% smaller than regular pickle!

3. API Response Optimization

from toonstream import encode
from flask import Flask, Response

app = Flask(__name__)

@app.route('/api/employees')
def get_employees():
    employees = db.query("SELECT * FROM employees")
    toon_data = encode(employees)
    return Response(toon_data, mimetype='text/plain')

# Clients get 55% smaller responses

4. Configuration Files

import toonstream

config = {
    "database": {"host": "localhost", "port": 5432},
    "cache": {"ttl": 3600, "max_size": 1000}
}

# Save human-readable config
with open('config.toon', 'w') as f:
    f.write(toonstream.encode(config, indent=2))

# Load config
with open('config.toon') as f:
    config = toonstream.decode(f.read())

---

🛠️ API Reference

Core Functions

encode(obj, format='toon', auto_mode=False, compact=False, smart_optimize=True, indent=None, sort_keys=False)

Convert Python object to TOON or TRON format.

Parameters:

• obj (Any): Python object (dict, list, primitive)
• format (str): Output format - 'toon' (default) or 'tron'. New in v2.0.0!
• auto_mode (bool): Auto-detect mode (tensor vs normal). (default: False)
• compact (bool): Minimize whitespace (default: False)
• smart_optimize (bool): Auto-detect best format (default: True)
• indent (int): Indentation spaces, None for compact (default: None)
• sort_keys (bool): Sort dictionary keys alphabetically (default: False)

Returns: str - TOON or TRON formatted string

Raises: ToonEncodeError - If encoding fails

# Basic encoding (TOON format - default)
toon = encode(data)

# TRON format - ultra-compact (New in v2.0.0!)
tron = encode(data, format='tron')

# Auto mode - automatically detects and handles tensors
toon = encode(data, auto_mode=True)

# Compact output
toon = encode(data, compact=True)

# Sort dictionary keys
toon = encode(data, sort_keys=True)

# Always use tabular (no optimization)
toon = encode(data, smart_optimize=False)

# Pretty print with 2-space indent
toon = encode(data, indent=2)

# Combine parameters
toon = encode(data, auto_mode=True, compact=True, sort_keys=True)

decode(data_str, format='toon', auto_mode=False, strict=True)

Convert TOON or TRON format to Python object.

Parameters:

• data_str (str): TOON or TRON formatted string
• format (str): Input format - 'toon' (default), 'tron', or 'auto'. New in v2.0.0!
• auto_mode (bool): Auto-detect mode for decoding. (default: False)
• strict (bool): Enforce strict validation (default: True)

Returns: Any - Python object

Raises: ToonDecodeError or TronDecodeError - If decoding fails

# Decode TOON string (default)
data = decode(toon_str)

# Decode TRON string (New in v2.0.0!)
data = decode(tron_str, format='tron')

# Auto mode - automatically detects and reconstructs tensors
data = decode(toon_str, auto_mode=True)

# Lenient mode (allows minor format issues)
data = decode(toon_str, strict=False)

# Combine parameters
data = decode(toon_str, auto_mode=True, strict=True)

TRON-Specific Functions (New in v2.0.0!)

tron_encode(obj)

Encode Python object directly to TRON format.

from toonstream import tron_encode

tron = tron_encode({"name": "Alice", "age": 30})
# Output: name=Alice;age=30

tron_decode(tron_str)

Decode TRON string directly to Python object.

from toonstream import tron_decode

data = tron_decode("name=Alice;age=30")
# Output: {'name': 'Alice', 'age': 30}

Pickle Functions

save_toon_pickle(data, filepath, smart_optimize=True, protocol=HIGHEST_PROTOCOL)

Save data as TOON-encoded pickle file.

Parameters:

• data (Any): Python object to save
• filepath (str): Output file path
• smart_optimize (bool): Use TOON optimization (default: True)
• protocol (int): Pickle protocol version (default: HIGHEST_PROTOCOL)

from toonstream import save_toon_pickle

save_toon_pickle(data, 'data.toon.pkl')

load_toon_pickle(filepath, strict=True)

Load TOON-encoded pickle file.

Parameters:

• filepath (str): Input file path
• strict (bool): Enforce strict TOON validation (default: True)

Returns: Any - Loaded Python object

from toonstream import load_toon_pickle

data = load_toon_pickle('data.toon.pkl')

Exceptions

• ToonError - Base exception
• ToonEncodeError - Encoding failures (unsupported types, NaN, Infinity)
• ToonDecodeError - Decoding failures (invalid format, syntax errors)
• ToonValidationError - Validation failures
• ToonPickleError - Pickle operation failures

---

🧪 Development & Testing

Running Tests

# Install dev dependencies
pip install -e ".[dev]"

# Run all tests (130 tests, all passing)
pytest tests/ -v

# Run specific test file
pytest tests/test_both_modes.py -v

# Run with coverage
pytest tests/ --cov=toonstream --cov-report=html

# Open coverage report
open htmlcov/index.html

Running Benchmarks

# Run all benchmarks
python benchmarks/run_all_comparisons.py

# Results appear in terminal and save to results/

Project Structure

toonstream/
├── toonstream/               # Core library
│   ├── __init__.py           # Public API exports
│   ├── encoder.py            # TOON encoder
│   ├── decoder.py            # TOON decoder
│   ├── tron_encoder.py       # TRON encoder (NEW v2.0.0)
│   ├── tron_decoder.py       # TRON decoder (NEW v2.0.0)
│   ├── tensor_utils.py       # PyTorch tensor support
│   ├── pickle_utils.py       # Pickle integration
│   ├── exceptions.py         # Exception hierarchy
│   └── unified_api.py        # Unified encode/decode with format parameter
├── benchmarks/               # Performance benchmarks
│   ├── run_all_comparisons.py
│   ├── compare_tron_formats.py  # 4-format comparison (NEW v2.0.0)
│   ├── complex_benchmarks.py    # 45 test cases (NEW v2.0.0)
│   ├── token_counters.py        # tiktoken integration (NEW v2.0.0)
│   └── config.json
├── tests/                    # Test suite (186 tests, 100% passing)
│   ├── test_toonstream.py    # Core functionality
│   ├── test_tron.py          # TRON format tests (56 tests - NEW v2.0.0)
│   └── ...
├── examples/                 # Usage examples
│   ├── basic_example.py      # Simple encoding/decoding
│   ├── tron_example.py       # TRON format usage (NEW v2.0.0)
│   ├── tron_tutorial.ipynb   # Interactive TRON tutorial (NEW v2.0.0)
│   ├── tensor_example.py     # PyTorch integration
│   └── README.md
├── .github/workflows/        # CI/CD workflows
│   ├── tests.yml             # Automated testing
│   ├── publish.yml           # Release & PyPI publishing
│   └── release-checklist.yml # Pre-release validation
├── data/                     # Benchmark datasets
├── results/                  # Benchmark results
├── README.md                 # This file
├── PICKLE_USAGE.md           # Pickle utilities guide
├── pyproject.toml            # Modern package configuration
├── setup.py                  # Package configuration
└── requirements.txt          # Dependencies

---

📖 Examples

See the examples/ directory for complete examples:

• basic_example.py - Getting started guide
• tron_example.py - Using TRON format (NEW in v2.0.0)
• tron_tutorial.ipynb - Interactive TRON tutorial (NEW in v2.0.0)
• tensor_example.py - PyTorch tensor integration
• README.md - Examples documentation

Run them:

python examples/basic_example.py
python examples/tron_example.py
python examples/tensor_example.py  # Requires PyTorch

What's New in v2.0.0?

TRON Format (Token Reduced Object Notation):

• 🚀 Ultra-compact serialization reducing token usage by 50-70%.
• 📊 65% savings vs JSON for typical datasets.
• 🔄 100% loss-less round-trip conversion.

New CLI Tool:

• 🖥️ Process files directly: toonstream encode data.json -o data.tron
• 🛠️ Easy integration into data pipelines.

Production Reliability:

• ✅ Expanded test suite (186 tests, 100% coverage).
• 🔒 Full type safety with strict mypy compliance.
• ⚡ Sub-millisecond performance.

Key Features:

• tron_encode() / tron_decode() specific APIs.
• Tabular optimization for arrays (@col1,col2|val1,val2).
• Dot notation for nested structures (key.subkey=value).
• Full backward compatibility with v1.1.0.

Previous: v1.1.0 (Auto Mode):

• Single auto_mode parameter (simpler API)
• Automatic tensor mode detection
• 130 tests, all passing

---

🤝 Contributing

Contributions welcome! Areas for improvement:

1. Additional Features - Streaming encoder, additional format options
2. Performance - C extension for faster encoding/decoding
3. Documentation - More examples, integration guides
4. Language Bindings - JavaScript, Go, Rust implementations

Development Setup

# Fork and clone
git clone https://github.com/vivekpandian08/toonstream.git
cd toonstream

# Create branch
git checkout -b feature/your-feature

# Install dev dependencies
pip install -e ".[dev]"

# Install pre-commit hooks
pre-commit install

# Make changes and test
pytest tests/

# Submit PR

---

📄 License

MIT License - see LICENSE file

---

🙏 Acknowledgments

• Inspired by CSV efficiency for tabular data
• Built for the LLM era where tokens = money
• Tested with real-world production datasets

---

📞 Support

• Issues: GitHub Issues
• Discussions: GitHub Discussions
• Documentation: See PICKLE_USAGE.md and results/OPTIMIZATION_GUIDE.md

---

🔗 Links

• PyPI: https://pypi.org/project/toonstream/
• GitHub: https://github.com/vivekpandian08/toonstream
• Repository: https://github.com/vivekpandian08/toonstream
• Issues: https://github.com/vivekpandian08/toonstream/issues

---

Made with ❤️ for the LLM community

Save tokens. Save money. Build better.