Proven compression, not projections

All three results use exact o200k_base counts in balanced mode. The JSON-data results are lossless. The schema benchmark preserves required fields and descriptions while trimming redundant examples. Repetitive, structured data benefits most.

What it does

• CLI — tokenfold compress / inspect files or stdin, and compare payloads with tokenfold diff; tokenfold wrap -- <command> compresses a command's output in place.
• Library — the same Rust engine and receipt shape from pip install tokenfold or npm install tokenfold, so behavior never drifts between languages.
• Proxy — tokenfold-proxy sits in front of your provider, compresses requests, and streams responses through untouched.
• MCP server — tokenfold mcp serve exposes compress/inspect/retrieve/stats to any MCP-compatible agent or editor.
• Receipts, not guesses — every call returns exact token counts, the transforms it applied, and any warnings, so you can audit what changed.
• Local and lightweight — one static Rust binary runs on your machine or infrastructure, with no hosted service or additional data processor.

Quick start

Install the interface that fits your stack:

pip install tokenfold       # Python 3.9+
npm install tokenfold       # Node.js 22+
cargo add tokenfold-core    # Rust library
cargo install tokenfold-cli # Rust CLI

Or download the CLI for Linux, macOS, or Windows from GitHub Releases, then verify it with the adjacent .sha256 file.

Preview the savings without changing the input, then write the compressed payload when you are ready:

tokenfold inspect payload.json --format json
tokenfold compress payload.json --format json --output payload.compact.json

Compress an OpenAI-style request before sending it to your provider:

import json
from pathlib import Path

from tokenfold import CompressionMode, compress_openai_payload

result = compress_openai_payload(
    Path("request.json").read_text(),
    mode=CompressionMode.BALANCED,
)
compressed_request = json.loads(result.payload)

print(f"saved {result.report.saved_tokens} tokens ({result.saved_pct():.1f}%)")
# Pass compressed_request to your existing OpenAI client.

The TypeScript package calls the same local Rust engine and returns bytes plus the canonical compression receipt:

import { compress } from "tokenfold";

const input = new TextEncoder().encode(JSON.stringify({
  results: [
    { id: 101, region: "us-east-1", plan: "pro" },
    { id: 102, region: "us-east-1", plan: "pro" },
    { id: 103, region: "us-east-1", plan: "pro" },
  ],
}, null, 2));

const { payload, report } = await compress(input, {
  format: "json",
  mode: "balanced",
});

console.log(`saved ${report.saved_tokens} tokens`);
console.log(new TextDecoder().decode(payload));

Want to try the CLI from source? Inspect the bundled request without changing it:

git clone https://github.com/snchimata/tokenfold.git
cd tokenfold
cargo run --release --locked -p tokenfold-cli -- \
  inspect examples/openai_payload.json --format openai

Across 100 requests with the same payload shape, those savings add up to:

json_minify          34,600 → 22,900   saved 11,700
schema_compaction    22,900 → 21,300   saved  1,600
TOTAL                34,600 → 21,300   saved 13,300 (38.4% reduction, estimated)

Why tokenfold

Models do not need the same object key hundreds of times. Providers still count every token. Tokenfold removes that structural waste before the model call, so you get:

• Lower input cost — send fewer billable tokens without changing providers.
• More useful context — reclaim room for instructions, evidence, and conversation history.
• Less data movement — shrink payloads crossing queues, proxies, logs, and evaluation runs.
• Fewer blind spots — inspect counts, transforms, and warnings.
• No new data processor — run locally, in-process, or behind your own loopback proxy.

messages · schemas · JSON · logs · diffs
                    │
                    ▼
                tokenfold ──────▶ any LLM provider
                    │
                    └───────────▶ compressed payload + receipt

When to use tokenfold · when to look elsewhere

Good fit if you...

• want an exact, auditable receipt for every call instead of an estimate
• need lossless guarantees on structured data — JSON, schemas, tool arguments — before it reaches a billing meter
• want a portable static binary with deterministic behavior and a small operational footprint
• are shrinking logs, diffs, and repetitive tool output before they hit an LLM, a log store, or a queue

Look elsewhere if you...

• need query-aware summarization of unstructured prose — that calls for a model, and tokenfold stays deterministic on purpose
• want a hosted API to call — tokenfold runs entirely on your machine or your infrastructure; there is nothing to sign up for

What it improves

Pick your integration

One Rust engine powers every surface, so policies and receipts stay consistent as your stack changes.

Compress generic JSON

Use format="JSON" for API responses, record dumps, and other data that is not an LLM request:

import json
import tokenfold

result = tokenfold.compress(
    json.dumps({
        "results": [
            {"id": 101, "region": "us-east-1", "plan": "pro"},
            {"id": 102, "region": "us-east-1", "plan": "pro"},
            {"id": 103, "region": "us-east-1", "plan": "pro"},
        ]
    }),
    format="JSON",
    mode="BALANCED",
)

print(f"saved {result.report.saved_tokens} tokens")

Run the proxy

cargo build --release --locked -p tokenfold-proxy
target/release/tokenfold-proxy \
  --upstream https://api.openai.com \
  --target-tokens 12000

The proxy listens on 127.0.0.1:8787 by default, streams SSE responses, and returns the compression receipt in X-TokenFold-* headers.

Safety you can inspect

Tokenfold recounts after every stage and stops when the target is met or the allowed transform set is exhausted.

• Never larger: a transform stays only when it reduces the token count.
• Reversible JSON: every structural rewrite must pass an exact round trip.
• Clear provenance: exact tokenizer results and estimates are labeled separately.
• Actionable receipts: every result lists savings, transforms, warnings, and final status.
• Honest limits: unreachable targets return an explicit status instead of silently deleting more content.

Lossy log and diff transforms remain policy-gated. Optional originals can be stored by SHA-256 hash; detected secret-shaped content is excluded.

Reproduce the numbers

Run the regression benchmark:

cargo bench -p tokenfold-core

Or inspect the small bundled JSON sample:

cargo run --release --locked -p tokenfold-cli -- \
  inspect examples/api_response.json --format json

The sample reports 382 → 206 estimated tokens, a 46.1% reduction. Ragged or compact inputs may save little; Tokenfold reports that result honestly.

Tokenfold Select

When structural compression ends, rank what matters.

Tokenfold Core removes structural waste. Tokenfold Select is its optional, query-aware companion: a LoRA-fine-tuned model that ranks text spans by relevance before you assemble a smaller context.

Select is deliberately separate from the CLI and libraries. That separation lets each tool do one job well: Core provides deterministic transforms and auditable receipts; Select adds query-aware ranking when relevance matters. Your allocator still owns required-content retention and the hard token budget.

from pathlib import Path

import torch
from huggingface_hub import snapshot_download
from peft import PeftModel
from transformers import AutoModelForSequenceClassification, AutoTokenizer

base_id = "ibm-granite/granite-embedding-reranker-english-r2"
repo_dir = Path(snapshot_download("snchimata/tokenfold-select"))
adapter_dir = repo_dir / "adapter"
tok = AutoTokenizer.from_pretrained(adapter_dir)
base = AutoModelForSequenceClassification.from_pretrained(base_id, dtype=torch.float32)
model = PeftModel.from_pretrained(base, adapter_dir).eval()

def score(query: str, spans: list[str]) -> list[float]:
    if not spans:
        return []
    enc = tok([query] * len(spans), spans, padding=True, truncation=True,
              max_length=8192, return_tensors="pt")
    with torch.no_grad():
        out = model(input_ids=enc["input_ids"], attention_mask=enc["attention_mask"])
        return out.logits.view(-1).float().tolist()

See the Tokenfold Select model card for setup, evaluation, training data, and limitations.

Contributing

Issues and pull requests are welcome. Run the core checks before opening a PR:

cargo fmt --all --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace --locked
python eval/run_fidelity.py --gate --profile smoke-first-consumer
cd packages/tokenfold && npm ci && npm test

License

Apache-2.0

Reclaim your context window

Start with one representative payload. Install tokenfold, inspect the receipt, and see how many tokens your application can stop sending today.

pip install tokenfold

If tokenfold earns a place in your stack, a ⭐ on GitHub helps the next team find it.