python-toon

Token-Oriented Object Notation for Python

A compact data format optimized for transmitting structured information to Large Language Models (LLMs) with 30-60% fewer tokens than JSON.

What is TOON?

TOON (Token-Oriented Object Notation) combines YAML's indentation-based structure for nested objects and CSV's tabular format for uniform data rows, optimized specifically for token efficiency in LLM contexts.

This is a faithful Python port of the original TOON TypeScript library by Johann Schopplich, maintaining 100% output compatibility with the official TOON specification.

Key Features

• 30-60% token reduction compared to standard JSON
• Minimal syntax: Eliminates redundant punctuation (braces, brackets, most quotes)
• Tabular arrays: CSV-like row format for uniform object collections
• Explicit metadata: Array length indicators [N] for validation
• LLM-friendly: Maintains semantic clarity while reducing token count
• 100% compatible with original TypeScript implementation

Installation

pip install python-toon

Quick Start

from toon import encode

# Simple object
data = {"name": "Alice", "age": 30}
print(encode(data))
# Output:
# name: Alice
# age: 30

# Tabular array (uniform objects)
users = [
    {"id": 1, "name": "Alice", "age": 30},
    {"id": 2, "name": "Bob", "age": 25},
    {"id": 3, "name": "Charlie", "age": 35},
]
print(encode(users))
# Output:
# [3,]{id,name,age}:
#   1,Alice,30
#   2,Bob,25
#   3,Charlie,35

# Complex nested structure
data = {
    "metadata": {"version": 1, "author": "test"},
    "items": [
        {"id": 1, "name": "Item1"},
        {"id": 2, "name": "Item2"},
    ],
    "tags": ["alpha", "beta", "gamma"],
}
print(encode(data))
# Output:
# metadata:
#   version: 1
#   author: test
# items[2,]{id,name}:
#   1,Item1
#   2,Item2
# tags[3]: alpha,beta,gamma

API Reference

encode(value, options=None)

Converts a Python value to TOON format.

Parameters:

• value (Any): JSON-serializable value to encode
• options (dict, optional): Encoding options

Returns: str - TOON-formatted string

Encoding Options

from toon import encode

encode(data, {
    "indent": 2,           # Spaces per indentation level (default: 2)
    "delimiter": ",",      # Delimiter for arrays: "," | "\t" | "|" (default: ",")
    "lengthMarker": "#"    # Optional marker prefix: "#" | False (default: False)
})

Delimiter Options

You can use string literals directly:

data = [1, 2, 3, 4, 5]

# Comma (default)
print(encode(data))
# [5]: 1,2,3,4,5

# Tab
print(encode(data, {"delimiter": "\t"}))
# [5	]: 1	2	3	4	5

# Pipe
print(encode(data, {"delimiter": "|"}))
# [5|]: 1|2|3|4|5

Or use the string keys:

encode(data, {"delimiter": "comma"})   # Default
encode(data, {"delimiter": "tab"})     # Tab-separated
encode(data, {"delimiter": "pipe"})    # Pipe-separated

Length Markers

Add the # prefix to array length indicators:

users = [
    {"id": 1, "name": "Alice"},
    {"id": 2, "name": "Bob"},
]

# Without marker (default)
print(encode(users))
# [2,]{id,name}:
# 1,Alice
# 2,Bob

# With marker
print(encode(users, {"lengthMarker": "#"}))
# [#2,]{id,name}:
#   1,Alice
#   2,Bob

Format Rules

Objects

Key-value pairs with primitives or nested structures:

{"name": "Alice", "age": 30}
# =>
# name: Alice
# age: 30

Primitive Arrays

Arrays always include length [N]:

[1, 2, 3, 4, 5]
# => [5]: 1,2,3,4,5

["alpha", "beta", "gamma"]
# => [3]: alpha,beta,gamma

Tabular Arrays

Uniform objects with identical primitive-only fields use CSV-like format:

[
    {"id": 1, "name": "Alice"},
    {"id": 2, "name": "Bob"},
]
# =>
# [2,]{id,name}:
#   1,Alice
#   2,Bob

Note: The delimiter appears in the length bracket [2,] for tabular arrays.

Mixed Arrays

Non-uniform data using list format with - markers:

[{"name": "Alice"}, 42, "hello"]
# =>
# [3]:
#   - name: Alice
#   - 42
#   - hello

Array Length Format

The length bracket format depends on the array type:

Tabular arrays (with fields):

• Delimiter always shown: [2,]{fields}: or [2|]{fields}: or [2\t]{fields}:

Primitive arrays (no fields):

• Comma: [3]: (delimiter hidden)
• Other: [3|]: or [3\t]: (delimiter shown)

Quoting Rules

Strings are quoted only when necessary (following the TOON specification):

• Empty strings
• Keywords: null, true, false
• Numeric strings: 42, -3.14
• Leading or trailing whitespace
• Contains structural characters: :, [, ], {, }, -, "
• Contains current delimiter (,, |, or tab)
• Contains control characters (newline, carriage return, tab, backslash)

"hello"          # => hello (no quotes)
"hello world"    # => hello world (internal spaces OK)
" hello"         # => " hello" (leading space requires quotes)
"null"           # => "null" (keyword)
"42"             # => "42" (looks like number)
""               # => "" (empty)

Type Conversions

Non-JSON types are normalized automatically:

• Numbers: Decimal form (no scientific notation)
• Dates/DateTime: ISO 8601 strings (quoted)
• Decimal: Converted to float
• Infinity/NaN: Converted to null
• Functions/Callables: Converted to null
• -0: Normalized to 0

LLM Integration Best Practices

When using TOON with LLMs:

1. Wrap in code blocks for clarity:

   ```toon
   name: Alice
   age: 30
   ```
   

2. Instruct the model about the format:

   "Respond using TOON format (Token-Oriented Object Notation). Use key: value syntax, indentation for nesting, and tabular format [N,]{fields}: for uniform arrays."

3. Leverage length markers for validation:

   encode(data, {"lengthMarker": "#"})
   

   Tell the model: "Array lengths are marked with [#N]. Ensure your response matches these counts."

4. Acknowledge tokenizer variance: Token savings depend on the specific tokenizer and model being used.

Token Efficiency Example

import json
from toon import encode

data = {
    "users": [
        {"id": 1, "name": "Alice", "age": 30, "active": True},
        {"id": 2, "name": "Bob", "age": 25, "active": True},
        {"id": 3, "name": "Charlie", "age": 35, "active": False},
    ]
}

json_str = json.dumps(data)
toon_str = encode(data)

print(f"JSON: {len(json_str)} characters")
print(f"TOON: {len(toon_str)} characters")
print(f"Reduction: {100 * (1 - len(toon_str) / len(json_str)):.1f}%")

# Output:
# JSON: 177 characters
# TOON: 85 characters
# Reduction: 52.0%

JSON output:

{"users": [{"id": 1, "name": "Alice", "age": 30, "active": true}, {"id": 2, "name": "Bob", "age": 25, "active": true}, {"id": 3, "name": "Charlie", "age": 35, "active": false}]}

TOON output:

users[3,]{id,name,age,active}:
  1,Alice,30,true
  2,Bob,25,true
  3,Charlie,35,false

Development

Setup

# Clone the repository
git clone https://github.com/xaviviro/python-toon.git
cd python-toon

# Create virtual environment
python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install in development mode
pip install -e .

# Install development dependencies
pip install -r requirements-dev.txt

Running Tests

# Run all tests
pytest

# Run with coverage
pytest --cov=pytoon --cov-report=term

# Run original compatibility tests
python test_original_cases.py

Type Checking

mypy src/pytoon

Linting

ruff check src/pytoon tests

Credits

This project is a Python implementation of the original TOON format created by Johann Schopplich.

Original TOON (TypeScript): MIT License © 2025-PRESENT Johann Schopplich

python-toon (Python port): Developed by Xavi Vinaixa

License

MIT License - see LICENSE file for details

Related

• Original TOON (TypeScript) - The original implementation
• TOON Format Specification (SPEC.md) - Official v1 specification with normative encoding rules
• TOON README - Format overview and examples

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

When contributing, please:

• Run python test_original_cases.py to ensure 100% compatibility with the original
• Add tests for new features
• Update documentation as needed

Support

For bugs and feature requests, please open an issue.