Skip to main content

Token Diet

token-diet

The Structural Context & Invisible Waste Auditor for AI Coding Agents

CI Release Homebrew Python Dependencies License: MIT

Audit token footprint, detect hidden context leaks, and prevent token-budget regressions across AI coding workflows.

Zero-dependency CLI for Claude Code, Cursor, Codex, Gemini CLI, OpenCode, and Windsurf.


Why Token Diet?

AI coding assistants accumulate hidden context overhead through oversized instruction directives (.cursorrules, AGENTS.md, CLAUDE.md), declared but unused MCP tool schemas, and unmetered git hooks that trigger cloud LLMs on every local commit.

This structural bloat reduces available working context for actual code and diagnostics, introduces unnecessary idle instructions into every prompt turn, and creates costly re-ingestion exposure upon prompt cache TTL expiration.

token-diet audits these structural sources directly on your local filesystem before tokens are loaded, serialized, or billed.


Real Terminal Audit in Action

Running token-diet delivers an instant diagnostic with zero runtime LLM overhead (< 10 ms execution):

Token Diet Real Terminal Audit

================================================================================
TOKEN DIET: FOOTPRINT & CONTEXT AUDIT
================================================================================
File / Directive                               |     Bytes |    Tokens |   Status
--------------------------------------------------------------------------------
AGENTS.md                                      |     2,450 |       612 |     [OK]
.cursorrules                                   |    18,240 |     4,560 |  [ALERT]
================================================================================
Total Bytes:     20,690 B
Total Tokens:    ~5,172 estimated base tokens per turn
Files Detected:  2 of 2

⚠️  HIGH CONTEXT FOOTPRINT DETECTED:
  Files marked [ALERT] exceed 16 KB (>4k tokens), increasing cache-miss exposure and context footprint.
  Recommendation: Decouple procedures into On-Demand Skills using the LUT Router pattern.

Quick 30-Second Install

Compatibility: Requires Python >= 3.10 (python3 --version) and standard library only (0 external dependencies). Compatible with macOS, Linux, and Windows.

Homebrew (macOS & Linux)

brew install joseoper/tap/token-diet

pipx (Isolated CLI) / pip

pipx install token-diet-cli
# Or:
pip install token-diet-cli

Run Directly from Source (No Installation Needed)

git clone https://github.com/joseoper/token-diet.git
cd token-diet
PYTHONPATH=src python3 -m token_diet

What It Does

token-diet organizes context defense across four core responsibilities:

Capability Scope & Mechanism Actionable Outcome
Audit Static filesystem inspection of directives (AGENTS.md, .cursorrules), global defaults (--global), MCP schemas (--mcp), and section breakdowns (--sections). Complete inventory of baseline context injection costs per turn.
Detect Heuristic flags identifying oversized rules (> 16 KB) and silent AI review invocations in .git/hooks/pre-commit. Instant visual badges ([OK], [WARN], [ALERT], 🚨 HOOK ALERT).
Enforce Deterministic quality gates for CI pipelines: strict limits (--strict), token ceilings (--max-tokens), and regression budgets (--baseline, --max-delta). Blocks pull requests and commits that introduce unbudgeted prompt expansion.
Recommend Architectural guidance: the Look-Up Table (LUT) Router pattern and on-demand skill decoupling (skills/<name>/SKILL.md). Projected base context reduction of ~91% while keeping rules accessible on demand.

Auto-Fix Rogue Git Hooks (--fix-hooks)

Neutralize unmonitored AI review commands in .git/hooks/pre-commit surgically: replaces offending subcommands on compound lines with true (preserving test runners like npm test or pytest), comments out standalone invocations, creates an atomic .bak backup, and preserves executable permissions:

# Preview changes with a colored unified diff:
token-diet --fix-hooks --dry-run

# Run with interactive confirmation [y/N]:
token-diet --fix-hooks

# Run non-interactively (ideal for automation and CI):
token-diet --fix-hooks -y

Why Not Use an LLM Gateway or Compressor?

  • Runtime Gateways (LiteLLM, Portkey): Operate downstream as network proxies. They are blind to local git hooks, uncommitted directives, or MCP schemas. By the time a gateway sees the payload, tokens have already been read from disk, serialized, and billed.
  • Text Compressors (LLMLingua): Use small models to drop tokens dynamically at runtime. This introduces inference latency, consumes GPU/CPU cycles, and risks corrupting code syntax, indentation, and architectural constraints.
  • Token Diet: Operates at rest as a static architectural auditor and deterministic CI gatekeeper. Zero dependencies, zero runtime LLM overhead (< 10 ms), and leaves application code completely untouched.

Documentation

Explore the complete documentation for advanced workflows, architecture, and CI integration:

  • 📖 CLI Usage Guide: Complete command options, flags, section breakdown, and JSON output reference.
  • 🏗️ Technical Architecture: Internal module design, hook isolation, static analysis, and boundaries.
  • 📊 Benchmarks & Methodology: Simulated 20-turn session model, prompt caching reality, and comparison matrix.
  • 🛠️ Remediation Playbook: Step-by-step runbook for the Look-Up Table (LUT) Router pattern.
  • 🚀 CI/CD Integration: GitHub Actions workflows and pre-commit gatekeeper examples.

Contributing

Contributions, issues, and feature requests are welcome. Review CONTRIBUTING.md for local setup, development guidelines, and pull request workflows:

# Run the test suite (131 deterministic tests)
python3 -m unittest discover -s tests -p 'test_*.py'

License

Distributed under the MIT License. Copyright © 2026 Token Diet Core Team.

Metadata

Release files for token-diet-cli 0.3.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for token-diet-cli 0.3.2
File Size Uploaded Provenance Actions
token_diet_cli-0.3.2.tar.gz 40.8 kB Without provenance Details for token_diet_cli-0.3.2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for token-diet-cli 0.3.2
File Interpreter ABI Platform Actions
token_diet_cli-0.3.2-py3-none-any.whl Python 3 none any Details for token_diet_cli-0.3.2-py3-none-any.whl

Total release size: 66.7 kB

Release files / token_diet_cli-0.3.2.tar.gz

Download URL token_diet_cli-0.3.2.tar.gz
Size 40.8 kB
Tags Source
SHA-256 checksum
How to use checksums
f3956c17b1b2f29851d93ddc285eb1f176d80e24abd60ee989096aa0b21bc946
BLAKE2b-256 checksum
How to use checksums
8133c8f9746829f7c3703c5e57c2859731cf7b88c36b012cef9861d7f1ebf012
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Release files / token_diet_cli-0.3.2-py3-none-any.whl

Download URL token_diet_cli-0.3.2-py3-none-any.whl
Size 25.9 kB
Tags Python 3
SHA-256 checksum
How to use checksums
ca4af1197f8c04fce8da3b999cdb5007018091fcd2c4d48dcb269d86c402f6bf
BLAKE2b-256 checksum
How to use checksums
de2fc339dad97a55df261f10b67aea2928380403683a16505ad434521f0f6c72
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.8 {"installer":{"name":"uv","version":"0.13.8","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Security for token-diet-cli 0.3.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page