token-diet

Compress prompts to use fewer tokens without losing meaning.

token-diet is a CLI tool that takes any prompt — from a file, stdin, or a pipe — and rewrites it to be as compact as possible while preserving the original intent. It works offline using battle-tested rewrite rules, or via the Claude API (claude-haiku-4-5-20251001) for intelligent AI-powered compression.

Quick install

pip install token-diet

Or from source:

git clone https://github.com/your-handle/token-diet
cd token-diet
pip install -e .

Usage

# From a file
token-diet prompt.txt

# From stdin
echo "Please summarize the following text for me..." | token-diet

# With Claude API key (intelligent compression)
token-diet prompt.txt --api-key sk-ant-...

# Or set the env var once
export ANTHROPIC_API_KEY=sk-ant-...
token-diet prompt.txt

# Compression levels
token-diet prompt.txt --level aggressive   # max compression, may lose nuance
token-diet prompt.txt --level balanced     # default, preserves intent
token-diet prompt.txt --level gentle       # minimal, safest changes only

# Show a diff between original and compressed
token-diet prompt.txt --diff

# Output ONLY the compressed prompt (great for piping)
token-diet prompt.txt --quiet | pbcopy
token-diet prompt.txt --quiet > compressed.txt

Example output

╭──────────────────────────────────────────╮
│            TOKEN DIET REPORT             │
╰──────────────────────────────────────────╯

Mode: balanced compression (rule-based / offline)

────────────────────── BEFORE ──────────────────────
│ Please take the following document and create a   │
│ comprehensive summary that covers all of the main │
│ points and key details. I would like you to make  │
│ sure to include every important fact...           │

────────────────────── AFTER ───────────────────────
│ Take the following document and create a thorough │
│ summary covering all main points and key details. │
│ Include every important fact...                   │

────────────────────── Results ─────────────────────
  Original:   847 tokens  (~3,388 chars)
  Compressed: 234 tokens  (~936 chars)
  Saved:      613 tokens  (72.4% reduction)
  Est. savings: $0.0031/call  (at GPT-4o pricing $5/1M tokens)

────────────────────── Techniques applied ──────────
  • Removed filler phrases (8)
  • Shortened verbose constructions (12)
  • Collapsed redundant instructions (3)

How it works

Offline (no API key)

The rule engine applies linguistically safe substitutions in three tiers:

With API key

Sends your prompt to claude-haiku-4-5-20251001 with a meta-prompt asking it to rewrite for minimum tokens. Falls back gracefully to rule-based compression on any API error.

Token counting

Uses a simple 4 chars ≈ 1 token heuristic by default. Install tiktoken for accurate OpenAI tokeniser counts:

pip install "token-diet[tiktoken]"

Options

Cost estimate

The report shows an estimated cost saving per API call based on GPT-4o input pricing ($5 / 1M tokens). This is illustrative — actual savings depend on which model you use and your actual call volume.

Development

# Clone and install in dev mode
git clone https://github.com/your-handle/token-diet
cd token-diet
python -m venv venv && source venv/bin/activate
pip install -e ".[dev]"

# Run tests
pytest -v

# Run tests with coverage
pytest --cov=src --cov-report=term-missing

# Lint and format
black src/ tests/
isort src/ tests/
flake8 src/ tests/
mypy src/

License

MIT