njft 0.1.0

A tool to convert complex, nested JSON into a token-efficient, flat TOON format.

NJFT: Nested JSON to TOON Converter

A Python package and command-line tool that converts complex JSON into a highly token-efficient, flat format called "TOON" (Tab-Object-Oriented Notation).

Achieve a 30% to 60% reduction in tokens when sending structured data to Large Language Models (LLMs), leading to lower API costs and faster responses.

🚀 Why NJFT? The Token Efficiency Advantage

Working with LLMs costs money, and every token counts. Standard JSON is verbose, filled with brackets, braces, and quotes that consume valuable tokens without adding much semantic value. NJFT was built to solve this problem.

By intelligently flattening the data structure and converting arrays of objects into a compact tabular format, NJFT dramatically reduces the character count of your data.

• 💰 Lower API Costs: Fewer tokens mean smaller bills from services like OpenAI, Anthropic, and Google.
• ⚡ Faster Performance: Less data needs to be sent and processed, resulting in quicker API responses.
• 📦 Fit More Data: Squeeze larger and more complex data structures into the LLM's context window.

Features

• Drastic Token Reduction: Achieves 30-60% character reduction compared to standard JSON.
• Flattens nested JSON objects.
• Converts arrays of objects into a special key[count]{headers}: format.
• Converts arrays of primitive values into indexed keys (e.g., key_0, key_1).
• Provides proxy metrics (character and word counts) to estimate token reduction.
• Command-line interface to read from files and write to files or standard output.

Requirements

• Python 3.6+
• No external libraries are needed.

Installation

You can install the package directly from GitHub using pip:

pip install njft

Usage

As a Command-Line Tool

Once installed, the njft command will be available in your terminal.

# 1. Basic Conversion (prints to console)
njft input.json

# 2. Save to an Output File
njft input.json -o output.toon

# 3. Display Conversion Metrics and Comparison
njft input.json --metrics

# 4. Use with Pipes (stdin and stdout)
cat input.json | njft - > output.toon

Arguments

• input_file: Path to the input JSON file. Use - to read from standard input.
• -o, --output: (Optional) Path to the output TOON file. If omitted, the output is printed to standard output.
• -m, --metrics: (Optional) If specified, the tool will display a detailed comparison of the input JSON and output TOON, including character/word counts and compression percentage.

As a Python Library

You can import and use the encode function in your own Python code.

Simple Example

import njft

# You can pass a file path
toon_from_file = njft.encode("path/to/your/input.json")
print(toon_from_file)

# Or you can pass a raw JSON string
json_string = '{"user": {"name": "John", "roles": ["admin", "editor"]}}'
toon_from_string = njft.encode(json_string)
print(toon_from_string)

Detailed Example with Complex JSON

Here is a more realistic example showing how njft simplifies a complex nested object. You can run this code by saving it as example.py.

import njft
import json

# 1. Define a complex, nested JSON object
complex_data = {
  "project_id": "P-487A",
  "manager": {
    "user_id": 101,
    "full_name": "Elias Vance"
  },
  "milestones": [
    {"name": "Phase 1", "status": "Completed", "tasks": [{"id": 1}, {"id": 2}]},
    {"name": "Phase 2", "status": "In Progress", "tasks": [{"id": 3}]}
  ]
}

# 2. Convert to a JSON string
input_json_string = json.dumps(complex_data)

# 3. Use njft.encode() to get the token-efficient TOON format
toon_output = njft.encode(input_json_string)

print(toon_output)

Example Conversion

Given the following input.json:

{
  "project": "TOON Converter",
  "milestones": [
    { "id": 1, "name": "Design", "status": "Completed" },
    { "id": 2, "name": "Implement", "status": "In Progress" }
  ]
}

Running python3 toonFlat.py input.json will produce:

project: TOON Converter
milestones{id,name,status}:
1,'Design','Completed'
2,'Implement','In Progress'