TokenPress

Blazing-fast LLM token compression engine, built in Rust.
2–5× fewer tokens. Same or better output quality. Any model, any language.

Why TokenPress?

LLM API calls are expensive. Most prompt tokens are redundant — filler words, predictable syntax, boilerplate. TokenPress uses information theory to score every token by its surprise value, keeps only what matters, and sends a compressed prompt to the LLM.

• 18.6 M tok/s Rust core (PyO3 + rayon)
• 3.3× compression with near-zero quality loss
• Drop-in wrappers for OpenAI & Anthropic — zero code changes
• 6 selection strategies: ratio, top-k, threshold, percentile, IQR, token merging
• Language & model agnostic — works on any text, any LLM

Install

pip install tokenpress              # core
pip install "tokenpress[models]"    # + scoring model (distilgpt2)
pip install "tokenpress[all]"       # + OpenAI & Anthropic wrappers

Quick Start

import tokenpress

result = tokenpress.compress("Your very long prompt …", ratio=0.3)
print(result.compressed_text)       # compressed prompt
print(f"{result.compression_ratio:.1f}× smaller, {result.savings_percentage:.0f}% tokens saved")

Drop-in OpenAI wrapper:

import openai, tokenpress

client = tokenpress.wrap(openai.OpenAI(), ratio=0.3)
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": very_long_prompt}],
)
# same API, 3× cheaper

CLI:

tokenpress compress document.txt -r 0.3 -o compressed.txt
tokenpress bench document.txt

How It Works

Each token is scored by self-information: $I(x_i) = -\log_2 P(x_i \mid x_{<i})$

High surprise → keep. Low surprise → remove. Pure math, no heuristics.

Built on research from Selective Context (EMNLP '23), TRIM (COLING '25), Token Merging (ICLR '23), H2O (NeurIPS '24), and LLMLingua-2 (ACL '24).

Links

• GitHub — full docs, benchmarks, architecture
• Benchmarks — 7 reproducible benchmark suites
• License — Apache-2.0