Measured results

Compression

All four use exact o200k_base counts in balanced mode. The 91.7% showcase keeps the planted 503 result inline and makes the other 98 long rows retrievable by hash. Run it with python examples/lossy_pruning.py.

Fine-tuned relevance ranking

Tokenfold Select is the optional query-aware model for choosing what fills a tight context window after structural compression ends.

At a 25% budget, the fine-tuned ranker keeps the answer 70% of the time and beats the strongest free heuristic by 15.8%. Critical-content survival is 100% at every measured budget through allocator force-keep. Results are three-seed repeated subsampling over roughly 73,000 training and 24,000 held-out fixtures per run; the model card publishes the training recipe and full baseline set. Here, task success means the literal gold answer survived selection under the stated budget.

One platform, two engines

Lossless or recoverable lossy

Quick start

Install the surface that fits your stack:

pip install tokenfold       # Python 3.9+
npm install tokenfold       # Node.js 22+
cargo add tokenfold-core    # Rust library
cargo install tokenfold-cli # Rust CLI

Or download a signed CLI build for Linux, macOS, or Windows from GitHub Releases and verify it with the adjacent .sha256 file.

Preview first, then compress:

tokenfold inspect payload.json --format json
tokenfold compress payload.json --format json --output payload.compact.json

Python uses the same Core engine and typed receipt:

import json
from pathlib import Path

from tokenfold import CompressionMode, compress_openai_payload

result = compress_openai_payload(
    Path("request.json").read_text(),
    mode=CompressionMode.BALANCED,
)
compressed_request = json.loads(result.payload)
print(f"saved {result.report.saved_tokens} tokens ({result.saved_pct():.1f}%)")

tokenfold init --agent <agent> installs a durable host integration; tokenfold doctor verifies it. Trusted filters for Git, build, and test output are available through tokenfold filters list.

Runnable examples:

python examples/quickstart.py
python examples/lossy_pruning.py

Recoverable lossy pruning

Lossless folding has a ceiling on heterogeneous arrays. --lossy ranks rows, keeps the strongest signals, and replaces selected rows with compact {"$tf_ref": {...}} handles. A row leaves the payload only after the local store accepts it.

tokenfold compress examples/incident_feed.json --format json \
  --lossy heuristic --lossy-ratio 0.35 --output feed.compact.json

On the bundled 40-event feed:

The planted 503 with success: false and retries: 7 survives every shown setting because typed failure signals outrank position and length. Long rows amortize marker overhead further: the 100-result showcase reaches 91.7%.

Fetch a dropped row:

tokenfold retrieve 978339a8898fedde3c5b0662a213f12ae4ad1b7fe6771f62b3c3d74d87389a4c
# {"seq":1,"ts":"2026-08-15T00:01:11Z","subsystem":"index-writer",...}

What the flags mean:

• --lossy-ratio is an aggression hint over eligible array items. Lower keeps fewer rows; it is not a whole-document guarantee.
• --target-tokens is the whole-document goal. Tokenfold stops when it reaches the target losslessly and reports best_effort when the safe transform set cannot reach it (unreachable_target when protected content alone exceeds the target).
• --lossy-preserve <path> protects a named array; nested paths conservatively protect their nearest eligible ancestor.
• Generic JSON only: lossy pruning does not run on OpenAI or Anthropic message payloads.
• Storage is fail-closed: refused rows stay inline, and detected secret-shaped bytes are never persisted.

Preview the projected savings with no store writes:

tokenfold compress examples/incident_feed.json --format json \
  --lossy heuristic --lossy-ratio 0.35 --dry-run

Tokenfold Select

When structure ends, rank what matters.

Tokenfold Select is an Apache-2.0 LoRA adapter on ibm-granite/granite-embedding-reranker-english-r2. It scores candidate spans against a query; your allocator applies the token budget and force-keeps required content. Core remains model-free and deterministic, while Select adds relevance when lexical heuristics stop being enough.

from pathlib import Path

import torch
from huggingface_hub import snapshot_download
from peft import PeftModel
from transformers import AutoModelForSequenceClassification, AutoTokenizer

base_id = "ibm-granite/granite-embedding-reranker-english-r2"
repo_dir = Path(snapshot_download("snchimata/tokenfold-select"))
adapter_dir = repo_dir / "adapter"
tokenizer = AutoTokenizer.from_pretrained(adapter_dir)
base = AutoModelForSequenceClassification.from_pretrained(
    base_id,
    dtype=torch.float32,
)
model = PeftModel.from_pretrained(base, adapter_dir).eval()

def score(query: str, spans: list[str]) -> list[float]:
    if not spans:
        return []
    encoded = tokenizer(
        [query] * len(spans),
        spans,
        padding=True,
        truncation=True,
        max_length=8192,
        return_tensors="pt",
    )
    with torch.no_grad():
        output = model(
            input_ids=encoded["input_ids"],
            attention_mask=encoded["attention_mask"],
        )
    return output.logits.view(-1).float().tolist()

See the Tokenfold Select model card for setup, evaluation, training data, and limitations.

Safety and auditability

• Never larger: Core keeps a transform only when exact recounting shows a reduction; a lossy branch must also beat the lossless result.
• Reversible structure: JSON folds must pass an exact round trip.
• Protected content: provider system messages and latest-user content are held behind format-aware safety gates.
• Clear provenance: exact tokenizer counts, heuristics, and extrapolations are labeled separately.
• Actionable receipts: every result lists savings, transforms, warnings, retrieval state, and final status.
• Local control: detected secrets are redacted before reports or storage; policy learning changes configuration only with --apply.

Reproduce the results

# 91.7% maximum-compression showcase, mixed-feed curve, retrieval, and preserve
python examples/lossy_pruning.py

# Lossless transform benchmarks
cargo bench -p tokenfold-core

# Small exact-token CLI example
cargo run --release --locked -p tokenfold-cli -- \
  inspect examples/api_response.json --format json

The small bundled API response reports 382 → 206 tokens, a 46.1% lossless reduction. Benchmark sources and thresholds live in CHANGELOG.md and crates/tokenfold-core/benches/THRESHOLDS.toml.

Contributing

Issues and pull requests are welcome. Run the relevant checks before opening a PR:

cargo fmt --all --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace --locked
python eval/run_fidelity.py --gate --profile smoke-first-consumer
cd packages/tokenfold && npm ci && npm test

License

Apache-2.0

Start with one representative payload, inspect the receipt, and see how many tokens your application can stop sending today.

pip install tokenfold

If Tokenfold earns a place in your stack, a ⭐ on GitHub helps the next team find it.