token-diet
The Structural Context & Invisible Waste Auditor for AI Coding Agents
Audit token footprint, detect hidden context leaks, and prevent token-budget regressions across AI coding workflows.
Zero-dependency CLI for Claude Code, Cursor, Codex, Gemini CLI, OpenCode, and Windsurf.
Why Token Diet?
AI coding assistants accumulate hidden context overhead through oversized instruction directives (.cursorrules, AGENTS.md, CLAUDE.md), declared but unused MCP tool schemas, and unmetered git hooks that trigger cloud LLMs on every local commit.
This structural bloat reduces available working context for actual code and diagnostics, introduces unnecessary idle instructions into every prompt turn, and creates costly re-ingestion exposure upon prompt cache TTL expiration.
token-diet audits these structural sources directly on your local filesystem before tokens are loaded, serialized, or billed.
Real Terminal Audit in Action
Running token-diet delivers an instant diagnostic with zero runtime LLM overhead (< 10 ms execution):
================================================================================
TOKEN DIET: FOOTPRINT & CONTEXT AUDIT
================================================================================
File / Directive | Bytes | Tokens | Status
--------------------------------------------------------------------------------
AGENTS.md | 2,450 | 612 | [OK]
.cursorrules | 18,240 | 4,560 | [ALERT]
================================================================================
Total Bytes: 20,690 B
Total Tokens: ~5,172 estimated base tokens per turn
Files Detected: 2 of 2
⚠️ HIGH CONTEXT FOOTPRINT DETECTED:
Files marked [ALERT] exceed 16 KB (>4k tokens), increasing cache-miss exposure and context footprint.
Recommendation: Decouple procedures into On-Demand Skills using the LUT Router pattern.
Quick 30-Second Install
Compatibility: Requires Python >= 3.10 (
python3 --version) and standard library only (0 external dependencies). Compatible with macOS, Linux, and Windows.
Homebrew (macOS & Linux)
brew install joseoper/tap/token-diet
pipx (Isolated CLI) / pip
pipx install token-diet-cli
# Or:
pip install token-diet-cli
Run Directly from Source (No Installation Needed)
git clone https://github.com/joseoper/token-diet.git
cd token-diet
PYTHONPATH=src python3 -m token_diet
What It Does
token-diet organizes context defense across four core responsibilities:
| Capability | Scope & Mechanism | Actionable Outcome |
|---|---|---|
| Audit | Static filesystem inspection of directives (AGENTS.md, .cursorrules), global defaults (--global), MCP schemas (--mcp), and section breakdowns (--sections). |
Complete inventory of baseline context injection costs per turn. |
| Detect | Heuristic flags identifying oversized rules (> 16 KB) and silent AI review invocations in .git/hooks/pre-commit. |
Instant visual badges ([OK], [WARN], [ALERT], 🚨 HOOK ALERT). |
| Enforce | Deterministic quality gates for CI pipelines: strict limits (--strict), token ceilings (--max-tokens), and regression budgets (--baseline, --max-delta). |
Blocks pull requests and commits that introduce unbudgeted prompt expansion. |
| Recommend | Architectural guidance: the Look-Up Table (LUT) Router pattern and on-demand skill decoupling (skills/<name>/SKILL.md). |
Projected base context reduction of ~91% while keeping rules accessible on demand. |
Auto-Fix Rogue Git Hooks (--fix-hooks)
Neutralize unmonitored AI review commands in .git/hooks/pre-commit surgically: replaces offending subcommands on compound lines with true (preserving test runners like npm test or pytest), comments out standalone invocations, creates an atomic .bak backup, and preserves executable permissions:
# Preview changes with a colored unified diff:
token-diet --fix-hooks --dry-run
# Run with interactive confirmation [y/N]:
token-diet --fix-hooks
# Run non-interactively (ideal for automation and CI):
token-diet --fix-hooks -y
Why Not Use an LLM Gateway or Compressor?
- Runtime Gateways (LiteLLM, Portkey): Operate downstream as network proxies. They are blind to local git hooks, uncommitted directives, or MCP schemas. By the time a gateway sees the payload, tokens have already been read from disk, serialized, and billed.
- Text Compressors (LLMLingua): Use small models to drop tokens dynamically at runtime. This introduces inference latency, consumes GPU/CPU cycles, and risks corrupting code syntax, indentation, and architectural constraints.
- Token Diet: Operates at rest as a static architectural auditor and deterministic CI gatekeeper. Zero dependencies, zero runtime LLM overhead (< 10 ms), and leaves application code completely untouched.
Documentation
Explore the complete documentation for advanced workflows, architecture, and CI integration:
- 📖 CLI Usage Guide: Complete command options, flags, section breakdown, and JSON output reference.
- 🏗️ Technical Architecture: Internal module design, hook isolation, static analysis, and boundaries.
- 📊 Benchmarks & Methodology: Simulated 20-turn session model, prompt caching reality, and comparison matrix.
- 🛠️ Remediation Playbook: Step-by-step runbook for the Look-Up Table (LUT) Router pattern.
- 🚀 CI/CD Integration: GitHub Actions workflows and pre-commit gatekeeper examples.
Contributing
Contributions, issues, and feature requests are welcome. Review CONTRIBUTING.md for local setup, development guidelines, and pull request workflows:
# Run the test suite (131 deterministic tests)
python3 -m unittest discover -s tests -p 'test_*.py'
License
Distributed under the MIT License. Copyright © 2026 Token Diet Core Team.
Metadata
Release files for token-diet-cli 0.3.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| token_diet_cli-0.3.2.tar.gz | 40.8 kB | Without provenance | Details for token_diet_cli-0.3.2.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| token_diet_cli-0.3.2-py3-none-any.whl | Python 3 | none | any | Details for token_diet_cli-0.3.2-py3-none-any.whl |
Total release size: 66.7 kB
Release files / token_diet_cli-0.3.2.tar.gz
| Download URL | token_diet_cli-0.3.2.tar.gz |
|---|---|
| Size | 40.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
f3956c17b1b2f29851d93ddc285eb1f176d80e24abd60ee989096aa0b21bc946
|
|
BLAKE2b-256 checksum How to use checksums |
8133c8f9746829f7c3703c5e57c2859731cf7b88c36b012cef9861d7f1ebf012
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Release files / token_diet_cli-0.3.2-py3-none-any.whl
| Download URL | token_diet_cli-0.3.2-py3-none-any.whl |
|---|---|
| Size | 25.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
ca4af1197f8c04fce8da3b999cdb5007018091fcd2c4d48dcb269d86c402f6bf
|
|
BLAKE2b-256 checksum How to use checksums |
de2fc339dad97a55df261f10b67aea2928380403683a16505ad434521f0f6c72
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Security for token-diet-cli 0.3.2
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.