tokenfold 0.3.2

Token-compression toolkit for LLM payloads (Python binding over the tokenfold-core Rust crate)

TOKENFOLD

Send less noise. Fit more context. Pay for fewer input tokens.

Local, provider-neutral compression for prompts, tool schemas, JSON, logs, and diffs.

Quick start · Why tokenfold · Integrations · Benchmarks

---

Proven compression, not projections

Repetitive JSON	API responses	Tool schemas
67.6% fewer tokens	61.3% fewer tokens	45.63% fewer tokens
50-record payload	30-record payload	1.8 MB OpenAI-style fixture

All three results use exact o200k_base counts in balanced mode. The JSON-data results are lossless. The schema benchmark preserves required fields and descriptions while trimming redundant examples. Repetitive, structured data benefits most.

Quick start

Install the interface that fits your stack:

pip install tokenfold       # Python 3.9+
npm install tokenfold       # Node.js 22+
cargo add tokenfold-core    # Rust library
cargo install tokenfold-cli # Rust CLI

Or download the v0.3.1 CLI for Linux, macOS, or Windows from GitHub Releases, then verify it with the adjacent .sha256 file.

Or install the Python package:

pip install tokenfold

Compress an OpenAI-style request before sending it to your provider:

import json
from pathlib import Path

from tokenfold import CompressionMode, compress_openai_payload

result = compress_openai_payload(
    Path("request.json").read_text(),
    mode=CompressionMode.BALANCED,
)
compressed_request = json.loads(result.payload)

print(f"saved {result.report.saved_tokens} tokens ({result.saved_pct():.1f}%)")
# Pass compressed_request to your existing OpenAI client.

The TypeScript package calls the same local Rust engine and returns bytes plus the canonical compression receipt:

import { compress } from "tokenfold";

const input = new TextEncoder().encode(JSON.stringify({
  results: [
    { id: 101, region: "us-east-1", plan: "pro" },
    { id: 102, region: "us-east-1", plan: "pro" },
    { id: 103, region: "us-east-1", plan: "pro" },
  ],
}, null, 2));

const { payload, report } = await compress(input, {
  format: "json",
  mode: "balanced",
});

console.log(`saved ${report.saved_tokens} tokens`);
console.log(new TextDecoder().decode(payload));

Want to try the CLI from source? Inspect the bundled request without changing it:

git clone https://github.com/snchimata/tokenfold.git
cd tokenfold
cargo run --release --locked -p tokenfold-cli -- \
  inspect examples/openai_payload.json --format openai

Across 100 requests with the same payload shape, those savings add up to:

json_minify          34,600 → 22,900   saved 11,700
schema_compaction    22,900 → 21,300   saved  1,600
TOTAL                34,600 → 21,300   saved 13,300 (38.4% reduction, estimated)

Why tokenfold

Models do not need the same object key hundreds of times. Providers still count every token. Tokenfold removes that structural waste before the model call, so you get:

• Lower input cost — send fewer billable tokens without changing providers.
• More useful context — reclaim room for instructions, evidence, and conversation history.
• Less data movement — shrink payloads crossing queues, proxies, logs, and evaluation runs.
• Fewer blind spots — inspect counts, transforms, and warnings.
• No new data processor — run locally, in-process, or behind your own loopback proxy.

messages · schemas · JSON · logs · diffs
                    │
                    ▼
                tokenfold ──────▶ any LLM provider
                    │
                    └───────────▶ compressed payload + receipt

What it improves

Workload	User benefit
APIs and record sets	Store repeated keys and values once
Provider requests	Shrink messages and schemas without changing API shape
Agent logs and diffs	Keep evidence; collapse repetitive output
Token budgets	Meet the target or return an honest best effort
Sensitive workflows	Redact detected secrets before reports or storage

Pick your integration

One Rust engine powers every surface, so policies and receipts stay consistent as your stack changes.

Surface	Best for	Install or run
Python	Applications and evaluation pipelines	pip install tokenfold
TypeScript	Node.js applications and automation	npm install tokenfold
Rust	Native embedding	cargo add tokenfold-core
CLI	Files and command output	Download a release binary
HTTP proxy	Provider-shaped traffic	Build tokenfold-proxy from source
MCP server	MCP-compatible agents and editors	tokenfold mcp serve

Compress generic JSON

Use format="JSON" for API responses, record dumps, and other data that is not an LLM request:

import json
import tokenfold

result = tokenfold.compress(
    json.dumps({
        "results": [
            {"id": 101, "region": "us-east-1", "plan": "pro"},
            {"id": 102, "region": "us-east-1", "plan": "pro"},
            {"id": 103, "region": "us-east-1", "plan": "pro"},
        ]
    }),
    format="JSON",
    mode="BALANCED",
)

print(f"saved {result.report.saved_tokens} tokens")

Run the proxy

cargo build --release --locked -p tokenfold-proxy
target/release/tokenfold-proxy \
  --upstream https://api.openai.com \
  --target-tokens 12000

The proxy listens on 127.0.0.1:8787 by default, streams SSE responses, and returns the compression receipt in X-TokenFold-* headers.

Safety you can inspect

Tokenfold recounts after every stage and stops when the target is met or the allowed transform set is exhausted.

• Never larger: a transform stays only when it reduces the token count.
• Reversible JSON: every structural rewrite must pass an exact round trip.
• Clear provenance: exact tokenizer results and estimates are labeled separately.
• Actionable receipts: every result lists savings, transforms, warnings, and final status.
• Honest limits: unreachable targets return an explicit status instead of silently deleting more content.

Lossy log and diff transforms remain policy-gated. Optional originals can be stored by SHA-256 hash; detected secret-shaped content is excluded.

Reproduce the numbers

Fixture	Exact token reduction	Source
Repetitive 50-record JSON	67.6%	Changelog
30-record API response	61.3%	Changelog
1.8 MB OpenAI tool schema	45.63%	Thresholds

Run the regression benchmark:

cargo bench -p tokenfold-core

Or inspect the small bundled JSON sample:

cargo run --release --locked -p tokenfold-cli -- \
  inspect examples/api_response.json --format json

The sample reports 382 → 206 estimated tokens, a 46.1% reduction. Ragged or compact inputs may save little; Tokenfold reports that result honestly.

Status

Version 0.3.1 is live on every supported distribution channel:

Ecosystem	Published packages	Artifacts
Python	tokenfold==0.3.1	Source distribution and ABI3 wheels for Linux x64/ARM64, macOS x64/ARM64, and Windows x64
TypeScript	tokenfold@0.3.1	Typed Node.js API plus the matching @tokenfold/cli-* native package, with no post-install download or local Rust build
Rust	tokenfold-core, tokenfold-output, tokenfold-learn, and tokenfold-cli	Source packages published in dependency order

GitHub release v0.3.1 provides SHA-256-checksummed (not signed) CLI binaries for Linux, macOS, and Windows, plus CycloneDX SBOMs.

Contributing

Issues and pull requests are welcome. Run the core checks before opening a PR:

cargo fmt --all --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace --locked
python eval/run_fidelity.py --gate --profile smoke-first-consumer
cd packages/tokenfold && npm ci && npm test

License

Apache-2.0

Reclaim your context window

Start with one representative payload. Install tokenfold, inspect the receipt, and see how many tokens your application can stop sending today.

pip install tokenfold