token-diet 0.1.0

Compress prompts to use fewer tokens without losing meaning

token-diet

Compress prompts to use fewer tokens without losing meaning.

token-diet is a CLI tool that takes any prompt — from a file, stdin, or a pipe — and rewrites it to be as compact as possible while preserving the original intent. It works offline using battle-tested rewrite rules, or via the Claude API (claude-haiku-4-5-20251001) for intelligent AI-powered compression.

Quick install

pip install token-diet

Or from source:

git clone https://github.com/your-handle/token-diet
cd token-diet
pip install -e .

Usage

# From a file
token-diet prompt.txt

# From stdin
echo "Please summarize the following text for me..." | token-diet

# With Claude API key (intelligent compression)
token-diet prompt.txt --api-key sk-ant-...

# Or set the env var once
export ANTHROPIC_API_KEY=sk-ant-...
token-diet prompt.txt

# Compression levels
token-diet prompt.txt --level aggressive   # max compression, may lose nuance
token-diet prompt.txt --level balanced     # default, preserves intent
token-diet prompt.txt --level gentle       # minimal, safest changes only

# Show a diff between original and compressed
token-diet prompt.txt --diff

# Output ONLY the compressed prompt (great for piping)
token-diet prompt.txt --quiet | pbcopy
token-diet prompt.txt --quiet > compressed.txt

Example output

╭──────────────────────────────────────────╮
│            TOKEN DIET REPORT             │
╰──────────────────────────────────────────╯

Mode: balanced compression (rule-based / offline)

────────────────────── BEFORE ──────────────────────
│ Please take the following document and create a   │
│ comprehensive summary that covers all of the main │
│ points and key details. I would like you to make  │
│ sure to include every important fact...           │

────────────────────── AFTER ───────────────────────
│ Take the following document and create a thorough │
│ summary covering all main points and key details. │
│ Include every important fact...                   │

────────────────────── Results ─────────────────────
  Original:   847 tokens  (~3,388 chars)
  Compressed: 234 tokens  (~936 chars)
  Saved:      613 tokens  (72.4% reduction)
  Est. savings: $0.0031/call  (at GPT-4o pricing $5/1M tokens)

────────────────────── Techniques applied ──────────
  • Removed filler phrases (8)
  • Shortened verbose constructions (12)
  • Collapsed redundant instructions (3)

How it works

Offline (no API key)

The rule engine applies linguistically safe substitutions in three tiers:

Level	What it removes
gentle	Politeness openers ("Please", "Could you please", "I would like you to"), obvious verbose forms ("in order to", "due to the fact that")
balanced	Everything in gentle, plus hedging adverbs ("very", "quite", "basically"), discourse markers ("of course", "clearly", "in fact"), and more verbose constructions
aggressive	Everything above, plus restructuring question openings ("Can you help me…"), discourse section headers, and more

With API key

Sends your prompt to claude-haiku-4-5-20251001 with a meta-prompt asking it to rewrite for minimum tokens. Falls back gracefully to rule-based compression on any API error.

Token counting

Uses a simple 4 chars ≈ 1 token heuristic by default. Install tiktoken for accurate OpenAI tokeniser counts:

pip install "token-diet[tiktoken]"

Options

Flag	Default	Description
--api-key KEY	$ANTHROPIC_API_KEY	Anthropic API key for Claude-powered compression
--level	balanced	Compression level: gentle, balanced, aggressive
--diff	off	Show unified diff between original and compressed
--quiet / -q	off	Output only the compressed text (no formatting)
--version		Show version and exit
--help / -h		Show help and exit

Cost estimate

The report shows an estimated cost saving per API call based on GPT-4o input pricing ($5 / 1M tokens). This is illustrative — actual savings depend on which model you use and your actual call volume.

Development

# Clone and install in dev mode
git clone https://github.com/your-handle/token-diet
cd token-diet
python -m venv venv && source venv/bin/activate
pip install -e ".[dev]"

# Run tests
pytest -v

# Run tests with coverage
pytest --cov=src --cov-report=term-missing

# Lint and format
black src/ tests/
isort src/ tests/
flake8 src/ tests/
mypy src/

License

MIT