tson 1.1.0

Token-efficient Structured Object Notation for LLMs

TSON for Python

Token-efficient Structured Object Notation - Python Implementation

Installation

cd python
pip install -e .

Or install from PyPI:

pip install tson

Quick Start

import tson

# Simple object
data = {"name": "Alice", "age": 30, "active": True}
encoded = tson.dumps(data)
print(encoded)
# Output: {@name,age,active|Alice,30,true}

# Perfect round-trip
decoded = tson.loads(encoded)
assert data == decoded  # True

API Reference

Serialization

import tson

# Serialize Python data to TSON string
tson_string = tson.dumps(data)

# Serialize to file
with open('data.tson', 'w') as f:
    tson.dump(data, f)

Deserialization

# Deserialize TSON string to Python data
data = tson.loads(tson_string)

# Deserialize from file
with open('data.tson', 'r') as f:
    data = tson.load(f)

Examples

1. Simple Objects

data = {"name": "Alice", "age": 30, "active": True}
encoded = tson.dumps(data)
# {@name,age,active|Alice,30,true}

2. Arrays

data = [1, 2, 3, 4, 5]
encoded = tson.dumps(data)
# [1,2,3,4,5]

3. Array of Objects (Tabular Format)

data = [
    {"id": 1, "name": "Alice", "email": "[e-mail]"},
    {"id": 2, "name": "Bob", "email": "[e-mail]"},
    {"id": 3, "name": "Carol", "email": "[e-mail]"}
]
encoded = tson.dumps(data)
# {@id,name,email#3|1,Alice,[e-mail]|2,Bob,[e-mail]|3,Carol,[e-mail]}

decoded = tson.loads(encoded)
assert data == decoded  # Perfect round-trip

4. Nested Schema Notation

data = [
    {"id": 1, "name": "Alice", "address": {"city": "NYC", "zip": "10001"}},
    {"id": 2, "name": "Bob", "address": {"city": "LA", "zip": "90001"}}
]
encoded = tson.dumps(data)
# {@id,name,address(@city,zip)#2|1,Alice,{NYC,"10001"}|2,Bob,{LA,"90001"}}

5. Complex Nested Structures

data = {
    "company": "Acme Corp",
    "employees": [
        {
            "id": 1,
            "name": "Alice",
            "skills": ["Python", "Go"],
            "contact": {"email": "[e-mail]", "phone": "555-0101"}
        },
        {
            "id": 2,
            "name": "Bob",
            "skills": ["Java"],
            "contact": {"email": "[e-mail]", "phone": "555-0102"}
        }
    ]
}

encoded = tson.dumps(data)
decoded = tson.loads(encoded)
assert data == decoded  # Perfect round-trip

6. Type Preservation

data = {
    "zip_string": "10001",  # String
    "zip_number": 10001,    # Number
    "version_string": "1.0", # String
    "version_number": 1.0   # Float
}

encoded = tson.dumps(data)
# {@zip_string,zip_number,version_string,version_number|"10001",10001,"1.0",1.0}

decoded = tson.loads(encoded)
assert isinstance(decoded["zip_string"], str)   # True
assert isinstance(decoded["zip_number"], int)   # True
assert isinstance(decoded["version_string"], str)  # True
assert isinstance(decoded["version_number"], float)  # True

7. Empty Values

data = {
    "empty_string": "",
    "empty_array": [],
    "empty_object": {},
    "null_value": None
}

encoded = tson.dumps(data)
# {@empty_string,empty_array,empty_object,null_value|"",[],{@},null}

8. Special Characters

data = {
    "comma": "hello, world",
    "pipe": "a|b|c",
    "quotes": 'She said "hello"',
    "newline": "line1\nline2"
}

encoded = tson.dumps(data)
decoded = tson.loads(encoded)
assert data == decoded  # Special characters preserved

Testing

Run the comprehensive test suite (14 tests):

python tests/test_round\ trip.py

All tests should pass:

[PASS] Simple object
[PASS] Simple array
[PASS] Array of objects (tabular)
[PASS] Nested object
[PASS] Mixed array
[PASS] Empty values
[PASS] Special characters
[PASS] Numeric strings
[PASS] Nested arrays
[PASS] Array with nested objects (nested schema)
[PASS] Complex structure
[PASS] Boolean values
[PASS] Numeric types

SUCCESS: All 13 tests passed!

Interactive Testing

Test TSON strings interactively:

python test_tson_string.py

This tool lets you:

• Enter TSON strings and see the JSON conversion
• Verify round-trip conversion
• See token savings
• Run example tests

LLM Integration

See llm_integration_example.py for complete examples of using TSON with LLMs.

Quick Example

import tson

# Prepare data for LLM
data = [
    {"date": "2025-01-01", "sales": 5000, "region": "North"},
    {"date": "2025-01-02", "sales": 6000, "region": "South"},
    {"date": "2025-01-03", "sales": 5500, "region": "East"},
]

tson_data = tson.dumps(data)

# Include TSON system prompt (see prompts.md)
system_prompt = """
TSON format (compact JSON):
• {@k1,k2|v1,v2} = object
• {@k1,k2#N|v1,v2|v1,v2} = array of objects
"""

user_prompt = f"Analyze this sales data: {tson_data}"

# Send to LLM API...
# Token savings: 30-50% compared to JSON

See ../prompts.md for complete LLM prompt templates.

Examples

Run the basic usage examples:

python examples/basic_usage.py

This demonstrates:

• Simple objects
• Array of objects (tabular format)
• Nested schema notation
• Mixed types
• Real-world data structures
• Special values

Type Support

TSON supports all JSON types:

Python Type	TSON Representation	Example
str	String (quoted if needed)	Alice or "Hello, World"
int	Number	42, -17
float	Number	3.14, -2.5
bool	Boolean	true, false
None	Null	null
list	Array	[1,2,3]
dict	Object	{@key|value}

Performance

TSON is optimized for:

• Token efficiency - 30-55% savings vs JSON
• Fast parsing - Simple delimiter-based parsing
• Low memory - Minimal overhead
• No dependencies - Pure Python implementation

Syntax Guide

See the main SPEC.md for complete syntax specification.

Quick reference:

Delimiter	Purpose	Example
{ }	Object boundaries	{@name|Alice}
[ ]	Array boundaries	[1,2,3]
@	Object marker	{@key1,key2|...}
,	Field/value separator	name,age,city
|	Row separator	val1,val2|val1,val2
#	Row count (optional)	#3

Requirements

• Python 3.7 or higher
• No external dependencies

Development

Install in editable mode for development:

pip install -e .

Run tests:

python tests/test_round\ trip.py

Contributing

Contributions to the Python implementation are welcome! Please see ../CONTRIBUTING.md for guidelines.

Documentation

• ../SPEC.md - Complete format specification
• QUICKSTART.md - Quick start guide
• ../prompts.md - LLM integration guide
• llm_integration_example.py - Working LLM examples

License

MIT License - see ../LICENSE file for details.

---

Version: 1.1.0 Status: Production Ready Python: 3.7+ Dependencies: None

Part of the TSON project by Zeno AI