toons 0.1.1

A high-performance TOON (Token Oriented Object Notation) parser and serializer for Python, implemented in Rust.

TOONS - Token Oriented Object Notation

A fast Rust-based parser and serializer for the TOON format (Token Oriented Object Notation), a token-efficient data serialization format designed specifically for Large Language Models.

This library provides a Python interface that mirrors the API of Python's standard json module, making it easy to work with TOON-formatted strings and files.

Architecture

The library is implemented in Rust using PyO3 for Python bindings, providing four core functions that parallel the standard library's json module:

• load(file) - Parse TOON data from a file object
• loads(string) - Parse TOON data from a string
• dump(obj, file) - Serialize Python object to TOON format and write to file
• dumps(obj) - Serialize Python object to TOON format string

The implementation follows the TOON Specification v1.3, ensuring proper support for:

• Indentation-based structure
• Array notation with element count (e.g., tags[3]: admin,ops,dev)
• Tabular format for uniform object arrays
• Configurable delimiters (comma, pipe, or tab)
• Unquoted keys and string values

Features

• Fast serialization/deserialization: Implemented in Rust with PyO3 bindings
• TOON Spec v1.3 compliant: Full support for the official TOON specification
• Token-efficient: 30-60% fewer tokens than equivalent JSON, ideal for LLM contexts
• Familiar API: Mirrors Python's json module interface (load, loads, dump, dumps)
• Python native types: Returns/accepts Python dict, list, str, int, float, bool, None
• File and string operations: Complete support for both file I/O and string operations

Installation

Development Installation

# Install development dependencies
pip install -e ".[dev]"

# Build the Rust extension
maturin develop

Production Installation

# Build wheel
maturin build --release

# Install the wheel
pip install target/wheels/*.whl

Usage

The API mirrors Python's json module for easy adoption:

String Operations

import toons

# Parse TOON string (loads)
toon_string = """
name: John Doe
age: 30
tags[3]: admin,developer,ops
"""
data = toons.loads(toon_string)
print(data)  # {'name': 'John Doe', 'age': 30, 'tags': ['admin', 'developer', 'ops']}

# Serialize to TOON string (dumps)
data = {
    "name": "John Doe",
    "age": 30,
    "tags": ["admin", "developer", "ops"]
}
toon_output = toons.dumps(data)
print(toon_output)
# Output:
# name: John Doe
# age: 30
# tags[3]: admin,developer,ops

File Operations

import toons

# Parse TOON file (load)
with open('data.toon', 'r') as f:
    data = toons.load(f)
    print(data)

# Serialize to TOON file (dump)
data = {"users": [{"name": "Alice", "age": 25}, {"name": "Bob", "age": 30}]}
with open('output.toon', 'w') as f:
    toons.dump(data, f)

TOON Format Examples

Simple object:

name: John
age: 30
active: true

Array notation:

tags[3]: admin,ops,dev

Nested structure:

user:
  name: John
  contacts:
    email: [e-mail]
    phone: [phone]

Tabular format (uniform objects):

users[2]{name,age}:
  Alice,25
  Bob,30

Supported Data Types

Python Type	TOON Format	Example
dict	Indented key-value pairs	name: John\nage: 30
list	Array notation with count	tags[3]: a,b,c
str	Unquoted (when safe)	name: John
int	Number literal	age: 30
float	Decimal literal	price: 19.99
bool	true/false	active: true
None	null	value: null

Note: The TOON format is significantly more token-efficient than JSON, especially for arrays and nested structures, making it ideal for LLM applications.

API Reference

loads(s: str) -> Any

Parse a TOON-formatted string and return the corresponding Python object.

Arguments:

• s (str): TOON-formatted string to parse

Returns:

• Python object (dict, list, str, int, float, bool, or None)

Raises:

• ValueError: If the string is not valid TOON format

load(fp: IO[str]) -> Any

Parse TOON data from a file object and return the corresponding Python object.

Arguments:

• fp: File-like object supporting .read()

Returns:

• Python object (dict, list, str, int, float, bool, or None)

Raises:

• ValueError: If the file content is not valid TOON format

dumps(obj: Any) -> str

Serialize a Python object to a TOON-formatted string.

Arguments:

• obj: Python object to serialize

Returns:

• TOON-formatted string

Raises:

• ValueError: If the object cannot be serialized to TOON format

dump(obj: Any, fp: IO[str]) -> None

Serialize a Python object to TOON format and write to a file object.

Arguments:

• obj: Python object to serialize
• fp: File-like object supporting .write()

Raises:

• ValueError: If the object cannot be serialized to TOON format

Error Handling

The library raises ValueError with descriptive error messages for invalid TOON syntax:

try:
    result = toons.loads('invalid toon syntax')
except ValueError as e:
    print(f"Parse error: {e}")

Examples

See the examples/ directory for simple usage examples:

# String operations (loads/dumps)
python examples/string_example.py

# File operations (load/dump)
python examples/file_example.py

Testing

Run the test suite with pytest:

# Run all tests
pytest

# Run with coverage
pytest --cov=toons

# Run specific test file
pytest tests/unit/test_loads.py

# Run specific test
pytest tests/unit/test_loads.py -k test_loads_simple_object

Development

Prerequisites

• Python 3.8+
• Rust (latest stable)
• maturin

Building

# Development build
maturin develop

# Release build
maturin build --release

Project Structure

toons/
├── src/
│   └── lib.rs              # Rust implementation
├── tests/
│   └── unit/               # Unit tests
│       ├── test_loads.py   # Tests for loads()
│       ├── test_dumps.py   # Tests for dumps()
│       ├── test_load.py    # Tests for load()
│       ├── test_dump.py    # Tests for dump()
│       └── test_roundtrip.py
├── examples/               # Usage examples
├── TOON_SPEC_1.3.md       # TOON specification reference
├── Cargo.toml             # Rust dependencies
├── pyproject.toml         # Python project config
└── README.md

TOON Format Specification

This library implements TOON Specification v1.3 (2025-10-31) using the rtoon Rust crate (v0.1.3).

Note: rtoon implements TOON Spec v1.2, which is fully compatible with v1.3. The specification is included in the repository as TOON_SPEC_1.3.md for reference.

Compliance Verification

The test suite in tests/unit/test_spec_compliance.py contains 40 tests verifying compliance with TOON Spec v1.3:

• ✅ All primitive types (null, bool, int, float, string)
• ✅ Object encoding with key-value pairs and 2-space indentation
• ✅ Array notation with element count [N]:
• ✅ Tabular format for uniform object arrays [N]{fields}:
• ✅ Nested structures and complex data
• ✅ Round-trip fidelity for all data types
• ✅ File I/O operations

Run tests: pytest tests/unit/test_spec_compliance.py -v

Why rtoon?

The rtoon crate was chosen as the implementation backend because it provides:

• Full round-trip support: Both encoding (serialization) and decoding (parsing) with complete fidelity
• TOON Spec v1.2 compliance: Follows the official TOON specification
• Robust parsing: Strict mode validation ensures data integrity
• Active maintenance: Updated regularly with the latest specification changes
• Well-tested: Comprehensive test suite ensures reliability

For more details on rtoon, see: https://github.com/shreyasbhat0/rtoon

License

This project is open source. See LICENSE file for details.