Measured results

Compression

All four use exact o200k_base counts in balanced preset. The 91.7% showcase keeps the planted 503 result inline and makes the other 98 long rows retrievable by hash. Run it with python examples/lossy_pruning.py. The versioned inputs and source provenance for these headline figures live in tests/fixtures/readme_metrics.json.

Fine-tuned relevance ranking

Tokenfold Select is an optional external query-aware model for choosing what fills a tight context window after structural compression ends. It is distributed and evaluated separately; Tokenfold Core does not invoke it.

At a 25% budget, the fine-tuned ranker keeps the answer 70% of the time and beats the strongest free heuristic by 15.8%. Critical-content survival is 100% at every measured budget through allocator force-keep. Results are three-seed repeated subsampling over roughly 73,000 training and 24,000 held-out fixtures per run; the model card publishes the training recipe and full baseline set. Here, task success means the literal gold answer survived selection under the stated budget. These are source-reported external results, pinned by model revision in the metric manifest; Core CI does not reproduce them.

One platform, two engines

Lossless or recoverable lossy

Quick start

Install the surface that fits your stack:

pip install tokenfold       # Python 3.9+
npm install tokenfold       # Node.js 22+
cargo add tokenfold-core    # Rust library
cargo install tokenfold-cli # Rust CLI

Or download a checksummed CLI build for Linux, macOS, or Windows from GitHub Releases and verify it with the adjacent .sha256 file.

Preview first, then compress:

tokenfold inspect payload.json --format json
tokenfold compress payload.json --format json --output payload.compact.json

Python uses the same Core engine and typed receipt:

import json
from pathlib import Path

from tokenfold import InputFormat, Preset, compress

result = compress(
    Path("request.json").read_bytes(),
    format=InputFormat.OPENAI_JSON,
    preset=Preset.BALANCED,
)
compressed_request = json.loads(result.payload)
print(f"saved {result.report.saved_tokens} tokens ({result.saved_pct():.1f}%)")

tokenfold init --agent claude-code merges a project-scoped .mcp.json entry without replacing other servers; tokenfold doctor --agent claude-code verifies it. Codex can use the same stdio server with codex mcp add tokenfold -- tokenfold mcp serve. See docs/configuration.md for tested MCP JSON/TOML and every environment override. Trusted filters for Git, build, and test output are available through tokenfold filters list.

Runnable examples, one per surface, all under examples/:

python examples/quickstart.py      # Python: compress messages, request bodies, JSON data
node examples/quickstart.mjs       # TypeScript/Node: compress, inspect, read the receipt
cargo run -p tokenfold-core --example quickstart   # Rust: the embedded core API
python examples/lossy_pruning.py   # CLI: opt-in recoverable pruning, end to end

examples/quickstart.ipynb is the guided tour of the whole Python surface: everything quickstart.py shows, plus previews, budgets and presets, provider payloads, recoverable pruning with retrieval, and where Tokenfold Select fits.

Recoverable lossy pruning

Lossless folding has a ceiling on heterogeneous arrays. --prune ranks rows, keeps the strongest signals, and replaces selected rows with compact {"$tf_ref": {...}} handles. A row leaves the payload only after the local store accepts it.

tokenfold compress examples/incident_feed.json --format json \
  --prune --keep-ratio 0.35 --output feed.compact.json

On the bundled 40-event feed:

The planted 503 with success: false and retries: 7 survives every shown setting because typed failure signals outrank position and length — at --keep-ratio 0.05 it is the only row kept. Long rows amortize marker overhead further: the 100-result showcase reaches 91.7%.

Fetch a dropped row:

tokenfold retrieve cb13cc59cca0c218c579cd1d4b3cbab58d6dea265eb995cc9c00faf0cd0a6856
# {"seq":1,"ts":"2026-08-15T00:01:11Z","subsystem":"index-writer",...}

What the flags mean:

• --keep-ratio is an aggression hint over eligible array items. Lower keeps fewer rows; it is not a whole-document guarantee.
• --target-tokens is the whole-document goal. Tokenfold stops when it reaches the target losslessly and reports best_effort when the safe transform set cannot reach it (unreachable when protected content alone exceeds the target).
• --preserve <path> protects a named array; nested paths conservatively protect their nearest eligible ancestor.
• Generic JSON only: lossy pruning does not run on OpenAI or Anthropic message payloads.
• Storage is fail-closed: refused rows stay inline, and detected secret-shaped bytes are never persisted.

Preview the projected savings with no store writes:

tokenfold inspect examples/incident_feed.json --format json \
  --prune --keep-ratio 0.35

The same flags, and the same fail-closed contract, from Python and TypeScript:

from tokenfold import InputFormat, PruningPolicy, compress, retrieve

pruning = PruningPolicy(keep_ratio=0.35)
result = compress(feed_bytes, format=InputFormat.JSON, pruning=pruning)
original = retrieve(marker["$tf_ref"])   # any dropped row, verbatim

import { compress, retrieve } from "tokenfold";

const { payload, report } = await compress(feed, {
  format: "json",
  pruning: { keepRatio: 0.35 },
});
const original = await retrieve(hash); // any dropped row, verbatim

Tokenfold Select

When structure ends, rank what matters.

Tokenfold Select is an Apache-2.0 LoRA adapter on ibm-granite/granite-embedding-reranker-english-r2. It scores candidate spans against a query; your allocator applies the token budget and force-keeps required content. It is a separately distributed companion: Core remains model-free and deterministic and neither loads nor invokes Select.

from pathlib import Path

import torch
from huggingface_hub import snapshot_download
from peft import PeftModel
from transformers import AutoModelForSequenceClassification, AutoTokenizer

base_id = "ibm-granite/granite-embedding-reranker-english-r2"
repo_dir = Path(snapshot_download("snchimata/tokenfold-select"))
adapter_dir = repo_dir / "adapter"
tokenizer = AutoTokenizer.from_pretrained(adapter_dir)
base = AutoModelForSequenceClassification.from_pretrained(
    base_id,
    dtype=torch.float32,
)
model = PeftModel.from_pretrained(base, adapter_dir).eval()

def score(query: str, spans: list[str]) -> list[float]:
    if not spans:
        return []
    encoded = tokenizer(
        [query] * len(spans),
        spans,
        padding=True,
        truncation=True,
        max_length=8192,
        return_tensors="pt",
    )
    with torch.no_grad():
        output = model(
            input_ids=encoded["input_ids"],
            attention_mask=encoded["attention_mask"],
        )
    return output.logits.view(-1).float().tolist()

See the Tokenfold Select model card for setup, evaluation, training data, and limitations.

Safety and auditability

• Never larger: Core keeps a transform only when exact recounting shows a reduction; a lossy branch must also beat the lossless result.
• Reversible structure: JSON folds must pass an exact round trip.
• Protected content: provider system messages and latest-user content are held behind format-aware safety gates.
• Clear provenance: exact tokenizer counts, heuristics, and extrapolations are labeled separately.
• Actionable receipts: every result lists savings, transforms, warnings, retrieval state, and final status.
• Local control: detected secrets are redacted before reports or storage; policy learning changes configuration only with --apply.

The offline fidelity gate uses deterministic lexical-overlap, critical-token, and containment proxies. Those checks catch regressions but do not establish semantic equivalence or downstream task success. Runtime quality fields are absent unless a versioned evaluator has supplied data; applications should run their own representative task evaluation before enabling lossy pruning.

Reproduce the results

# 91.7% maximum-compression showcase, mixed-feed curve, retrieval, and preserve
python examples/lossy_pruning.py

# Lossless transform benchmarks
cargo bench -p tokenfold-core

# Small exact-token CLI example
cargo run --release --locked -p tokenfold-cli -- \
  inspect examples/api_response.json --format json

The bundled 30-record API response reports 3,812 → 1,376 tokens, a 63.9% lossless reduction. Benchmark sources and thresholds live in CHANGELOG.md and crates/tokenfold-core/benches/THRESHOLDS.toml.

Contributing

Issues and pull requests are welcome. Run the relevant checks before opening a PR:

cargo fmt --all --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace --locked
python eval/run_fidelity.py --gate --profile smoke-first-consumer
cd packages/tokenfold && npm ci && npm test

License

Apache-2.0

Start with one representative payload, inspect the receipt, and see how many tokens your application can stop sending today.

pip install tokenfold

If Tokenfold earns a place in your stack, a ⭐ on GitHub helps the next team find it.