token-diet
Audit token footprint and detect silent context leaks in AI agent workflows.
Zero-dependency CLI for Claude Code, Cursor, Codex, Gemini, and OpenCode.
Why?
AI coding assistants inject instruction files (AGENTS.md, CLAUDE.md, .cursorrules) into the system prompt on every single turn. A bloated 25 KB prompt file forces the model to re-process ~6,000 tokens on every message, draining rate limits and breaking prompt cache discounts.
Worse, some local tools install hidden pre-commit hooks that send your entire git diff to an LLM on every commit.
token-diet scans your repository or workstation in milliseconds to:
- Audit prompt size: Flags instruction files that exceed context budgets (
[OK],[WARN],[ALERT]). - Catch rogue git hooks: Detects and disables silent LLM review hooks in the active hooks directory (
--fix-hooks), including linked worktrees, submodules, and repositories usingcore.hooksPath. - Locate the bloat: Breaks a Markdown directive down per level-2 heading so you know which section to trim (
--sections). - Audit MCP overhead: Reports the context cost of every declared MCP server in
opencode.json,.mcp.json, andmcp.json(--mcp). - Guard CI pipelines: Fails PR builds if prompt expansions exceed token thresholds or regress past a saved baseline (
--strict,--max-tokens,--baseline,--max-delta). - Zero dependencies: Pure Python standard library (Python >= 3.8). Runs in < 10ms.
Installation
Via Homebrew (macOS & Linux)
brew install joseoper/tap/token-diet
# Or:
brew tap joseoper/tap
brew install token-diet
Via pipx (Recommended for Isolated CLI)
pipx install token-diet-cli
Via pip / uv
# Using pip:
pip install token-diet-cli
# Using uv:
uv tool install token-diet-cli
Running Directly from Source (No Installation Needed)
git clone https://github.com/joseoper/token-diet.git
cd token-diet
PYTHONPATH=src python3 -m token_diet
Usage
1. Instant Health Check in Current Directory
Simply run token-diet in any project repository. It automatically scans local directives and checks for dangerous pre-commit hooks:
token-diet
Example Output:
================================================================================
TOKEN DIET: FOOTPRINT & CONTEXT AUDIT
================================================================================
File / Directive | Bytes | Tokens | Status
--------------------------------------------------------------------------------
/Users/developer/my-project/AGENTS.md | 962 | 240 | [OK]
/Users/developer/my-project/.cursorrules | 1,240 | 310 | [OK]
================================================================================
Total Bytes: 2,202 B
Total Tokens: ~550 estimated base tokens per turn
Files Detected: 2 of 2
2. Automated Alerts: Context Bloat & Invisible Pre-Commit Hooks
When a prompt file is oversized or a git hook silently burns tokens on every commit, token-diet issues an instant visual alert:
token-diet
Example Alert Output:
================================================================================
TOKEN DIET: FOOTPRINT & CONTEXT AUDIT
================================================================================
File / Directive | Bytes | Tokens | Status
--------------------------------------------------------------------------------
/Users/developer/my-project/AGENTS.md | 24,500 | 6,125 | [ALERT]
================================================================================
Total Bytes: 24,500 B
Total Tokens: ~6,125 estimated base tokens per turn
Files Detected: 1 of 1
🚨 IRREGULAR HOOK DETECTED:
• [pre-commit] Pre-commit hook invokes AI/LLM review tooling (gga, opencode). This causes invisible token consumption on every commit.
Path: /Users/developer/my-project/.git/hooks/pre-commit
💡 Tip: Run 'token-diet --fix-hooks' to automatically neutralize silent AI invocations.
⚠️ CONTEXT BLOAT DETECTED:
Files marked [ALERT] exceed 16 KB (>4k tokens) and severely penalize context windows.
Recommendation: Decouple procedures into On-Demand Skills using the LUT Router pattern.
3. Auto-Fix Silent Pre-Commit Hooks (--fix-hooks)
Neutralize unmonitored AI review tooling in .git/hooks/pre-commit in a single command with an automatic backup (.git/hooks/pre-commit.bak):
token-diet --fix-hooks
Example Output:
🛠️ AUTO-REMEDIATION: PRE-COMMIT HOOKS NEUTRALIZED
• Backup saved: /Users/developer/my-project/.git/hooks/pre-commit.bak
• Neutralized 1 AI review invocation(s).
• Safe pre-commit hook is active without silent token bleed.
4. Audit System-Wide Global Prompts (--global)
Scan default assistant instruction files across your entire workstation (~/.claude/CLAUDE.md, ~/.codex/AGENTS.md, ~/.gemini/GEMINI.md, etc.):
token-diet --global
5. Audit Specific Paths or Folders
# Audit a single file
token-diet AGENTS.md
# Audit entire directories recursively
token-diet docs/
6. Automated CI / Pre-Commit Gate
Integrate into your CI pipeline or Git pre-commit workflow to prevent prompt regressions:
# Fail CI if prompt footprint exceeds 2,500 tokens:
token-diet --max-tokens 2500 AGENTS.md
# Fail CI if any file triggers a [WARN] or [ALERT]:
token-diet --strict docs/
7. Structured Telemetry Output (--json)
Emit structured JSON for automated pipelines, threshold assertions, or telemetry dashboards:
token-diet --json
8. Locate Prompt Bloat per Heading (--sections)
A monolithic directive rarely bloats evenly. This breakdown attributes bytes and estimated tokens to every level-2 (##) heading, so you know exactly which section to compress or move into an on-demand skill:
token-diet --sections AGENTS.md
SECTION BREAKDOWN: MARKDOWN HEADING FOOTPRINT
================================================================================
AGENTS.md (23,914 B / ~5,978 tokens)
Line | Heading | Bytes | Tokens | Status
--------------------------------------------------------------------------------
1 | (preamble) | 412 | 103 | [OK]
12 | Core Directives | 19,220 | 4,805 | [ALERT]
98 | Additional Protocols | 4,282 | 1,070 | [WARN]
--------------------------------------------------------------------------------
Content before the first ## heading is grouped as (preamble), and deeper headings (###) stay attributed to their parent section. Combine with --json for machine-readable output.
9. Audit MCP Server Overhead (--mcp)
Every declared MCP server consumes context, and some configs inline large tool schemas. token-diet discovers local opencode.json, .mcp.json, mcp.json, .cursor/mcp.json, and .vscode/mcp.json files and reports the payload cost per server:
token-diet --mcp
MCP SERVER PAYLOAD AUDIT
================================================================================
opencode.json [opencode] (2 server(s))
Server | Transport | Tools | Bytes | Tokens | Status
--------------------------------------------------------------------------------
engram | local | 12 | 18,430 | 4,607 | [ALERT]
web-search | remote | 0 | 142 | 35 | [OK]
--------------------------------------------------------------------------------
Config-declared tools blocks are counted and included in each server's payload. Schemas fetched by a server at runtime are not visible to a local audit; the reported figure is the local injection cost floor.
10. Fail on Regression vs a Saved Baseline (--baseline, --max-delta)
Record a baseline once, then gate every pull request against it. This catches slow, incremental growth that a fixed threshold never notices:
# Save the baseline on your default branch
token-diet --json > token-baseline.json
# Fail if the total grows more than 5% in a pull request
token-diet --baseline token-baseline.json --max-delta 5
--baseline accepts any --json report ({"summary": {"total_tokens": N}}) or a minimal {"total_tokens": N} file. --max-delta exits with code 1 only when tokens increase beyond the given percentage; a decrease, an unchanged total, or growth exactly at the budget passes.
Health Status Indicators
| Indicator | Byte Threshold | Estimated Tokens | What it Means & Recommended Action |
|---|---|---|---|
🟢 [OK] |
< 4 KB |
< 1,000 tokens |
Optimal Context: Fits cleanly within prompt cache. Preserves fast, cheap inference. |
🟡 [WARN] |
4 KB – 16 KB |
1,000 – 4,000 tokens |
Moderate Size: Acceptable, but review for duplicate instructions or unneeded verbosity. |
🔴 [ALERT] |
> 16 KB |
> 4,000 tokens |
Severe Prompt Bloat: Monolithic rules degrade reasoning and waste context budget every turn. Decouple into On-Demand Skills. |
🚨 [HOOK ALERT] |
— | — | Hidden Token Bleed: The active pre-commit hook sends code to cloud LLMs on every git commit. Replace with static linters. |
What To Do When a Leak is Detected (Action Paths)
When token-diet flags an irregularity, follow the corresponding action path (detailed runbooks available in the Remediation Playbook):
Path A: When an Irregular Git Hook is Flagged (🚨 IRREGULAR HOOK DETECTED)
- Problem: A local hook inside
.git/hooks/pre-commitsecretly sends git diffs to cloud AI tools on every single commit. - Action Path:
- Inspect the script:
cat .git/hooks/pre-commit - Remove or comment out the automated AI review invocation.
- Replace it with fast, local deterministic linters (
ruff,eslint,biome). - Relocate semantic AI reviews to Pull Request gates in CI (see CI/CD Integration), where reviews are metered and visible.
- Inspect the script:
Path B: When an Agent Directive is Bloated (🔴 [ALERT])
- Problem: A root instruction file (
AGENTS.md,CLAUDE.md,.cursorrules) exceeds 16 KB (> 4,000 tokens), exhausting your token budget on every turn and destroying cache discounts. - Action Path (The Look-Up Table / LUT Router Pattern):
- Extract Specialized Protocols: Move detailed procedure manuals, test specifications, and review arbitrations into standalone on-demand skills (e.g.,
skills/<skill-name>/SKILL.md). - Compress Root Directive: Keep only core behavioral invariants and a routing index in the root file (< 3 KB / < 750 tokens):
## On-Demand Skills - When running database migrations, read `skills/db-migrate/SKILL.md`. - When preparing a release, read `skills/oss-publisher/SKILL.md`.
- Re-run
token-dietto confirm the status turns 🟢[OK].
- Extract Specialized Protocols: Move detailed procedure manuals, test specifications, and review arbitrations into standalone on-demand skills (e.g.,
Path C: When CI Fails on Token Budget Exceeded (exit code 1)
- Problem: A Pull Request introduced unbudgeted prompt expansion that violated
--max-tokens <N>or--strict. - Action Path:
- Run
token-diet --jsonto get the machine-readable breakdown of offending files. - Prune duplicate rules or extract text into skills.
- Re-run local check before pushing:
token-diet --strict.
- Run
Agentic Remediation: Prompt for AI Assistants
To have your AI coding assistant (Claude Code, Gemini CLI, Cursor, Windsurf, OpenCode) remediate prompt bloat autonomously without manual edits, copy and paste this prompt:
Run `token-diet` in this repository. If you detect any files marked [ALERT] or [WARN], follow `docs/remediation-playbook.md` to decouple operational procedures into on-demand skills under `skills/<skill-name>/SKILL.md`. Compress the root instruction file to a concise Look-Up Table (LUT Router) under 3 KB (< 750 tokens). If an irregular hook is flagged, run `token-diet --fix-hooks`. Verify that all tests pass and `token-diet` reports [OK].
Running Tests
Token Diet includes an automated test suite with deterministic coverage:
python3 -m unittest discover -s tests -p 'test_*.py'
Repository Structure
token-diet/
├── .github/
│ └── workflows/
│ └── ci.yml # Automated multi-version CI test runner
├── docs/
│ ├── remediation-playbook.md # Step-by-step token containment and LUT routing
│ └── ci-integration.md # CI workflows & pre-commit gatekeeper examples
├── src/
│ └── token_diet/ # Core Python package (Zero external dependencies)
│ ├── __init__.py
│ ├── __main__.py
│ ├── baseline.py # Baseline comparison & regression budget gate
│ ├── cli.py # CLI implementation and footprint engine
│ ├── git_hooks.py # Worktree, gitdir and core.hooksPath resolution
│ ├── mcp_audit.py # MCP config discovery & payload accounting
│ ├── metrics.py # Shared byte thresholds and token estimation
│ └── sections.py # Per-heading Markdown footprint breakdown
├── tests/
│ ├── test_baseline.py # Baseline budget unit tests
│ ├── test_git_hooks.py # Hook resolution unit tests
│ ├── test_mcp_audit.py # MCP payload unit tests
│ ├── test_sections.py # Section breakdown unit tests
│ └── test_token_audit.py # Core engine and CLI integration tests
├── .gitignore
├── AGENTS.md # Agent-friendly repository directives
├── CONTRIBUTING.md # Contribution guidelines & engineering principles
├── LICENSE # MIT Open Source License
├── pyproject.toml # PEP 621 packaging and CLI entrypoint
└── README.md # Project overview and documentation
Core Methodology: The 6 Vectors of Token Leakage
Token Diet is built on empirical analysis of how developer environments silently deplete API quotas:
- Global Prompt Bloat: Monolithic instruction files (
CLAUDE.md,AGENTS.md) exceeding 20 KB inject tens of thousands of static tokens on every turn. Solution: Keep root directives under 3 KB (<750 tokens) and route to on-demand skills. - Invisible Git Hooks: Pre-commit hooks that intercept local commits to invoke cloud LLM reviews drain quotas without developer consent. Solution: Keep local commits deterministic; reserve AI reviews for PR gates.
- Reasoning Effort Inflation: Running frontier models at
highreasoning effort for simple syntax or diagnostic queries generates thousands of hidden chain-of-thought tokens. Solution: Default tomediumorlow. - Static MCP Schemas: Registering dozens of MCP tools serializes massive JSON parameter schemas into system prompts. Solution: Prune unused tool servers and enforce lazy-loading.
- Prompt Cache Invalidation: Injecting dynamic timestamps or changing session IDs at the start of instructions destroys provider prompt caching discounts (up to 90%). Solution: Maintain immutable instruction prefixes.
- In-Session Runtime Inefficiencies: Agents loading entire 1,500-line files to check one function, overwriting whole files instead of focused diffs, or running uncompacted 100k+ token threads. Solution: Enforce active runtime hygiene (grep-first, line-range reads, targeted diffs, and context compaction thresholds; see Remediation Playbook Phase 4).
Platform Compatibility
Token Diet is OS-agnostic and cross-platform:
- CLI Diagnostic Tool: Pure Python standard library with zero third-party dependencies. Runs seamlessly on macOS, Linux, and Windows (Python >= 3.8).
- Universal Methodology: Core principles apply across all assistants (Claude Code, Gemini CLI, OpenAI Codex, OpenCode, Cursor, Windsurf).
Documentation & Guides
For in-depth operational procedures and automated pipeline setups:
- Remediation Playbook: Step-by-step runbook to prune prompt files, configure reasoning effort, and contain background services.
- CI/CD Integration: GitHub Actions workflows and pre-commit configurations for automated token budget enforcement.
- Agent Directives (AGENTS.md): Operational guidelines for AI coding assistants working on this codebase.
- Contributing Guidelines: How to run tests, submit pull requests, and coding standards.
License
This project is licensed under the terms of the MIT License.
Metadata
Release files for token-diet-cli 0.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| token_diet_cli-0.2.0.tar.gz | 36.0 kB | Without provenance | Details for token_diet_cli-0.2.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| token_diet_cli-0.2.0-py3-none-any.whl | Python 3 | none | any | Details for token_diet_cli-0.2.0-py3-none-any.whl |
Total release size: 59.2 kB
Release files / token_diet_cli-0.2.0.tar.gz
| Download URL | token_diet_cli-0.2.0.tar.gz |
|---|---|
| Size | 36.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
646a2e80ae8ef24b068148056d35c34046280f1678c44caa9c04e71f27edc675
|
|
BLAKE2b-256 checksum How to use checksums |
9c406be7f82e65f721f1034d4a6029ac21c898a0bf5c67137a17bec0fcf970b8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Release files / token_diet_cli-0.2.0-py3-none-any.whl
| Download URL | token_diet_cli-0.2.0-py3-none-any.whl |
|---|---|
| Size | 23.3 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
ea2564939446b486c6856141247675c64648844b3f33b14e317e1e284e6b3688
|
|
BLAKE2b-256 checksum How to use checksums |
570d00f8ab2a4c9e0cce77565ee66496b511c56753e9440e2b8364aaf124311b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
|
Security for token-diet-cli 0.2.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.