toon-parse 2.5.0

A powerful context reduction tool centered around converting data (JSON, YAML, XML, CSV) to the token-optimized TOON format for efficient LLM interactions.

🚀 TOON Converter (Python)

A powerful context reduction tool centered around converting data (JSON, YAML, XML, CSV) to TOON (Token-Oriented Object Notation) format for efficient LLM interactions.

Reduce your LLM token costs by up to 40% using the TOON format!

• Documentation: https://toonformatter.net/docs.html?package=toon-parse
• Source Code: https://github.com/ankitpal181/toon-formatter-py
• Bug Reports: https://github.com/ankitpal181/toon-formatter-py/issues
• POC Tool: https://toonformatter.net/

pip install toon-parse

🛠️ CLI Utility

Convert data and validate formats directly from your terminal using the unified toon-parse command.

Features

• Streamlined: Single command for all conversion types.
• Piping: Full support for stdin and stdout.
• Validation: Standalone format validation logic.
• Security: Built-in support for all encryption modes.

Usage Examples

# 1. Basic Conversion (JSON to TOON)
echo '{"name": "Alice"}' | toon-parse --from json --to toon

# 2. File-based Conversion with Async core
toon-parse --from xml --to json --input data.xml --output data.json --async

# 3. Secure Export (JSON to Encrypted XML)
toon-parse --from json --to xml --mode export --key <my_key> --input data.json

# 4. Format Validation
toon-parse --validate toon --input my_data.toon

🔄 Unified Format Converters

Beyond TOON, you can now convert directly between JSON, YAML, XML, and CSV using dedicated converter classes.

from toon_parse import JsonConverter, YamlConverter, XmlConverter, CsvConverter

# JSON <-> XML
xml_output = JsonConverter.to_xml({"user": "Alice"})
json_output = XmlConverter.to_json(xml_output)

# CSV <-> YAML
yaml_output = CsvConverter.to_yaml("id,name\n1,Alice")
csv_output = YamlConverter.to_csv(yaml_output)

Key Features

• Direct Conversion: No need to convert to TOON first.
• Mixed Text Support: from_json, from_xml, and from_csv methods automatically extract data from unstructured text.
• Return Types:
  • JsonConverter.from_toon and from_yaml support return_json=True (default) to return a JSON string, or False to return a dict/list.
  • YamlConverter.to_json supports return_json=True (default) to return a JSON string.
  • All other methods return strings (formatted xml, csv, yaml, etc.).

🔐 Using Encryption with Unified Converters

All new converters support the secure middleware pattern. Use the instance-based approach:

from toon_parse import JsonConverter, Encryptor

# 1. Setup Encryptor
enc = Encryptor(algorithm='fernet', key=my_key)

# 2. Initialize Converter with Encryptor
converter = JsonConverter(encryptor=enc)

# 3. Convert with security mode
# Example: Decrypt input JSON -> convert to XML -> Encrypt output
encrypted_xml = converter.to_xml(
    encrypted_json_input, 
    conversion_mode="middleware"
)

🚀 Quick Start

Basic Usage (Synchronous)

from toon_parse import ToonConverter

# 1. Python Object to TOON
data = {"name": "Alice", "age": 30, "active": True}
toon_string = ToonConverter.from_json(data)
print(toon_string)
# Output:
# name: "Alice"
# age: 30
# active: true

# 2. TOON to Python Object
json_output = ToonConverter.to_json(toon_string)
print(json_output)
# Output: {'name': 'Alice', 'age': 30, 'active': True}

Mixed Text Support

The library can automatically extract and convert JSON, XML, and CSV data embedded within normal text. This is perfect for processing LLM outputs.

from toon_parse import ToonConverter

# Text with embedded JSON
mixed_text = """
Here is the user profile you requested:
{
    "id": 101,
    "name": "Bob",
    "roles": ["admin", "editor"]
}
Please verify this information.
"""

# Automatically finds JSON, converts it to TOON, and preserves surrounding text
result = ToonConverter.from_json(mixed_text)
print(result)

# Output:
# Here is the user profile you requested:
# id: 101
# name: "Bob"
# roles[2]: "admin", "editor"
# Please verify this information.

🧠 Smart Code Optimization (New!)

The library includes an intelligent Data Manager that preprocesses input to handle code blocks efficiently.

• Code Preservation: Code snippets (detected via heuristics) are identified and protected from conversion logic.
• Context Reduction: Code blocks are automatically optimized to reduce token usage by:
  • Removing comments (# ..., // ...).
  • Compressing double newlines to single newlines.
  • Stripping unnecessary whitespace.

This ensures that while your data is converted to TOON for efficiency, any embedded code remains syntactically valid but token-optimized.

📉 Context Optimization (Expensive Words) (New!)

The library automatically identifies and replaces common "expensive" phrases with token-efficient alternatives to reduce the overall payload size required for LLM input.

Note: While most alterations significantly reduce token count, some replacements may only reduce character count while keeping the token count the same. This still helps in reducing the API payload size (in bytes), which can reduce latency and cost for bandwidth-constrained environments.

Examples:

• "large language model" → "llm"
• "frequently asked questions" → "faq"
• "as soon as possible" → "asap"
• "do not" → "don't"
• "I am" → "i'm"

This feature is case-insensitive and ensures that words inside code blocks and data blocks are NOT altered.

🔐 Secure Conversion Middleware

The ToonConverter can act as a secure middleware for processing encrypted data streams (e.g., from microservices). It handles the full Decrypt -> Convert -> Encrypt pipeline internally.

Supported Algorithms

• Fernet: High security (AES-128). Requires cryptography.
• XOR: Lightweight obfuscation.
• Base64: Encoding only.

Conversion Modes

1. "middleware": Encrypted Input → Encrypted Output (Decrypt → Convert → Re-encrypt)
2. "ingestion": Encrypted Input → Plain Output (Decrypt → Convert)
3. "export": Plain Input → Encrypted Output (Convert → Encrypt)
4. "no_encryption": Standard conversion (default)

Example Workflow

from toon_parse import ToonConverter, Encryptor
from cryptography.fernet import Fernet

# Setup
key = Fernet.generate_key()
enc = Encryptor(key=key, algorithm='fernet')
converter = ToonConverter(encryptor=enc)

# --- Mode 1: Middleware (Encrypted -> Encrypted) ---
raw_data = '{"user": "Alice", "role": "admin"}'
encrypted_input = enc.encrypt(raw_data)  # Simulate upstream encrypted data

# Converter decrypts, converts to TOON, and re-encrypts
encrypted_toon = converter.from_json(
    encrypted_input, 
    conversion_mode="middleware"
)
print(f"Secure Result: {encrypted_toon}")

# --- Mode 2: Ingestion (Encrypted -> Plain) ---
plain_toon = converter.from_json(
    encrypted_input,
    conversion_mode="ingestion"
)
print(f"Decrypted TOON: {plain_toon}")

# --- Mode 3: Export (Plain -> Encrypted) ---
my_data = {"status": "ok"}
secure_packet = converter.from_json(
    my_data,
    conversion_mode="export"
)
print(f"Encrypted Output: {secure_packet}")

⚡ Async Usage

For non-blocking operations in async applications (e.g., FastAPI), use AsyncToonConverter.

import asyncio
from toon_parse import AsyncToonConverter, Encryptor

async def main():
    # 1. Standard Async Usage
    converter = AsyncToonConverter()
    text = 'Data: <user><name>Alice</name></user>'
    toon = await converter.from_xml(text)
    print(toon)

    # 2. Async with Secure Middleware
    enc = Encryptor(algorithm='base64')
    secure_converter = AsyncToonConverter(encryptor=enc)
    
    # Decrypt -> Convert -> Encrypt (Middleware Mode)
    encrypted_msg = "eyJrZXkiOiAidmFsIn0=" # Base64 for {"key": "val"}
    
    # Use conversion_mode to specify pipeline behavior
    result = await secure_converter.from_json(
        encrypted_msg, 
        conversion_mode="middleware"
    )
    print(result)

asyncio.run(main())

📦 Batch Processing

Process multiple data items concurrently using batch converters. Available for all formats: TOON, JSON, YAML, XML, and CSV.

Synchronous Batch Converters

from toon_parse import BatchToonConverter, BatchJsonConverter, BatchYamlConverter

# Process multiple items in parallel
converter = BatchToonConverter()

json_list = [
    {"name": "Alice", "age": 30},
    {"name": "Bob", "age": 25},
    {"name": "Charlie", "age": 35}
]

# Convert all items to TOON (runs in parallel)
toon_results = converter.from_json(json_list, parallel=True)
print(len(toon_results))  # 3

# Convert back to JSON
json_results = converter.to_json(toon_results, parallel=True, return_json=False)
# Returns list of dicts

Available Batch Converters

All batch converters support the same methods as their non-batch counterparts:

from toon_parse import (
    BatchToonConverter,   # TOON ↔ JSON/YAML/XML/CSV
    BatchJsonConverter,   # JSON ↔ TOON/YAML/XML/CSV
    BatchYamlConverter,   # YAML ↔ TOON/JSON/XML/CSV
    BatchXmlConverter,    # XML ↔ TOON/JSON/YAML/CSV
    BatchCsvConverter     # CSV ↔ TOON/JSON/YAML/XML
)

# Example: Cross-format batch conversion
yaml_converter = BatchYamlConverter()
xml_converter = BatchXmlConverter()

yaml_list = ["name: Alice", "name: Bob", "name: Charlie"]

# YAML → XML (batch)
xml_results = yaml_converter.to_xml(yaml_list, parallel=True)

# XML → JSON (batch)
json_results = xml_converter.to_json(xml_results, parallel=True)

Batch Validation

from toon_parse import BatchToonConverter

# Validate multiple TOON strings
toon_list = [
    "name: Alice\nage: 30",
    "invalid toon",
    "name: Bob\nage: 25"
]

results = BatchToonConverter.validate(toon_list, parallel=True)

for i, result in enumerate(results):
    print(f"Item {i}: {'✅ Valid' if result['is_valid'] else '❌ Invalid'}")

Batch with Encryption

from toon_parse import BatchJsonConverter, Encryptor
from cryptography.fernet import Fernet

key = Fernet.generate_key()
enc = Encryptor(key=key, algorithm='fernet')
converter = BatchJsonConverter(encryptor=enc)

# Encrypt multiple JSON strings
json_list = ['{"id": 1}', '{"id": 2}', '{"id": 3}']
encrypted_inputs = [enc.encrypt(j) for j in json_list]

# Batch convert with middleware mode (decrypt → convert → encrypt)
encrypted_toon = converter.to_toon(
    encrypted_inputs,
    conversion_mode="middleware",
    parallel=True
)

⚡ Async Batch Processing

For high-performance non-blocking batch operations, use async batch converters. These use asyncio.gather for implicit parallelism.

Key Features

• Implicit Parallelism: No parallel flag needed - concurrency is automatic via asyncio.gather
• Non-blocking I/O: All file operations use thread pool executors
• Event Loop Friendly: Doesn't block the event loop during batch processing

Basic Usage

import asyncio
from toon_parse import AsyncBatchToonConverter, AsyncBatchJsonConverter

async def main():
    converter = AsyncBatchToonConverter()
    
    # Process 100 items concurrently
    json_list = [{"id": i, "name": f"User{i}"} for i in range(100)]
    
    # Automatically runs all conversions concurrently
    toon_results = await converter.from_json(json_list)
    
    print(f"Converted {len(toon_results)} items")

asyncio.run(main())

Available Async Batch Converters

from toon_parse import (
    AsyncBatchToonConverter,   # TOON ↔ JSON/YAML/XML/CSV
    AsyncBatchJsonConverter,   # JSON ↔ TOON/YAML/XML/CSV
    AsyncBatchYamlConverter,   # YAML ↔ TOON/JSON/XML/CSV
    AsyncBatchXmlConverter,    # XML ↔ TOON/JSON/YAML/CSV
    AsyncBatchCsvConverter     # CSV ↔ TOON/JSON/YAML/XML
)

Cross-Format Async Batch Pipeline

import asyncio
from toon_parse import AsyncBatchJsonConverter, AsyncBatchYamlConverter

async def convert_pipeline():
    json_conv = AsyncBatchJsonConverter()
    yaml_conv = AsyncBatchYamlConverter()
    
    # Start with 50 JSON objects
    json_data = [{"user": f"user{i}", "active": True} for i in range(50)]
    
    # JSON → YAML (concurrent)
    yaml_results = await json_conv.to_yaml(json_data)
    
    # YAML → TOON (concurrent)
    toon_results = await yaml_conv.to_toon(yaml_results)
    
    return toon_results

results = asyncio.run(convert_pipeline())

Concurrent Multi-Format Processing

import asyncio
from toon_parse import (
    AsyncBatchJsonConverter,
    AsyncBatchYamlConverter,
    AsyncBatchXmlConverter
)

async def process_all_formats():
    # Initialize converters
    json_conv = AsyncBatchJsonConverter()
    yaml_conv = AsyncBatchYamlConverter()
    xml_conv = AsyncBatchXmlConverter()
    
    # Different format data
    json_data = [{"id": 1}, {"id": 2}]
    yaml_data = ["name: Alice", "name: Bob"]
    xml_data = ["<user>Alice</user>", "<user>Bob</user>"]
    
    # Process all formats concurrently
    results = await asyncio.gather(
        json_conv.to_toon(json_data),
        yaml_conv.to_toon(yaml_data),
        xml_conv.to_toon(xml_data)
    )
    
    json_toon, yaml_toon, xml_toon = results
    return json_toon, yaml_toon, xml_toon

asyncio.run(process_all_formats())

Async Batch Validation

import asyncio
from toon_parse import AsyncBatchToonConverter

async def validate_batch():
    toon_list = [
        "name: Alice\nage: 30",
        "invalid toon data",
        "name: Bob\nage: 25"
    ]
    
    # Concurrent validation
    results = await AsyncBatchToonConverter.validate(toon_list)
    
    for i, result in enumerate(results):
        status = "✅ Valid" if result['is_valid'] else "❌ Invalid"
        print(f"Item {i}: {status}")

asyncio.run(validate_batch())

Async Batch with Encryption

import asyncio
from toon_parse import AsyncBatchJsonConverter, Encryptor
from cryptography.fernet import Fernet

async def secure_batch_processing():
    key = Fernet.generate_key()
    enc = Encryptor(key=key, algorithm='fernet')
    converter = AsyncBatchJsonConverter(encryptor=enc)
    
    # Prepare encrypted inputs
    json_list = ['{"user": "Alice"}', '{"user": "Bob"}', '{"user": "Charlie"}']
    encrypted_inputs = [enc.encrypt(j) for j in json_list]
    
    # Batch convert with middleware (decrypt → convert → encrypt)
    # All items processed concurrently
    encrypted_yaml = await converter.to_yaml(
        encrypted_inputs,
        conversion_mode="middleware"
    )
    
    return encrypted_yaml

asyncio.run(secure_batch_processing())

Performance Comparison

import asyncio
import time
from toon_parse import BatchToonConverter, AsyncBatchToonConverter

# Generate test data
data = [{"id": i, "value": f"test{i}"} for i in range(200)]

# Synchronous batch (with parallel=True)
start = time.perf_counter()
sync_converter = BatchToonConverter()
sync_results = sync_converter.from_json(data, parallel=True)
sync_time = time.perf_counter() - start

# Async batch (implicit parallelism)
async def async_convert():
    converter = AsyncBatchToonConverter()
    return await converter.from_json(data)

start = time.perf_counter()
async_results = asyncio.run(async_convert())
async_time = time.perf_counter() - start

print(f"Sync batch: {sync_time:.2f}s")
print(f"Async batch: {async_time:.2f}s")
# Async is typically 20-30% faster and doesn't block the event loop

📚 Features & Support

Feature	JSON	XML	CSV	YAML	TOON
Python Dict/List Input	✅	N/A	N/A	N/A	N/A
Pure String Input	✅	✅	✅	✅	✅
Mixed Text Support	✅	✅	✅	❌	❌
Async Support	✅	✅	✅	✅	✅
Encryption Support	✅	✅	✅	✅	✅

• Mixed Text: Finds occurrences of data formats in text (JSON, XML, CSV) and converts them in-place.
• Encryption: Supports Fernet, XOR, and Base64 middleware conversions.

⚙️ Static vs Instance Usage

Conversion Methods (from_json, to_json, etc.)

All conversion methods support both static and instance calling patterns:

from toon_parse import ToonConverter

# ✅ Static Usage (No Encryption)
toon = ToonConverter.from_json({"key": "value"})

# ✅ Instance Usage (Encryption Supported)
converter = ToonConverter(encryptor=enc)
toon = converter.from_json({"key": "value"}, conversion_mode="export")

Important:

• Static calls (ToonConverter.from_json(...)) work but cannot use encryption features.
• Instance calls are required to use conversion_mode and encryption middleware.

The same applies to async methods.

Validate Method

The validate() method is strictly static and does not support encryption:

# ✅ Correct Usage
result = ToonConverter.validate('key: "value"')

# ❌ Will NOT work with encryption
converter = ToonConverter(encryptor=enc)
result = converter.validate(encrypted_data)  # No decryption happens!

Why? Validation returns a dictionary (not a string), which cannot be encrypted. If you need to validate encrypted data, decrypt it first manually:

decrypted = enc.decrypt(encrypted_toon)
result = ToonConverter.validate(decrypted)

The same applies to AsyncToonConverter.validate().

🛠 API Reference

Core Converters

ToonConverter (Legacy & Easy Use)

• Static & Instance.
• Central hub for converting TOON <-> Any Format.

JsonConverter

• Focus: JSON <-> Any Format.
• from_toon(..., return_json=True)
• from_yaml(..., return_json=True)
• to_xml, to_csv, to_yaml, to_toon

YamlConverter

• Focus: YAML <-> Any Format.
• to_json(..., return_json=True)
• from_json, from_xml, from_csv, from_toon

XmlConverter

• Focus: XML <-> Any Format.
• to_json (returns JSON string), from_json, etc.

CsvConverter

• Focus: CSV <-> Any Format.
• to_json (returns JSON string), from_json, etc.

Note: All to_csv methods return a string. If the input is nested JSON/Object, it will be automatically flattened (e.g., user.name) to fit the CSV format. Conversely, from_csv will unflatten dotted keys back into objects.

Async Converters

Mirroring the synchronous classes, we have:

• AsyncJsonConverter
• AsyncYamlConverter
• AsyncXmlConverter
• AsyncCsvConverter

Usage is identical, just use await.

Encryption

Encryptor

Constructor: Encryptor(key=None, algorithm='fernet')

• algorithm: 'fernet' (default), 'xor', 'base64'.
• key: Required for Fernet/XOR.
• encrypt(data), decrypt(data): Helper methods.

Utility Functions

from toon_parse import extract_json_from_string, extract_xml_from_string, extract_csv_from_string
# Direct access to extraction logic without conversion

📄 License

MIT License