Skip to main content

token-diet

Audit token footprint and detect silent context leaks in AI agent workflows.

Zero-dependency CLI for Claude Code, Cursor, Codex, Gemini, and OpenCode.


Why?

AI coding assistants inject instruction files (AGENTS.md, CLAUDE.md, .cursorrules) into the system prompt on every single turn. A bloated 25 KB prompt file forces the model to re-process ~6,000 tokens on every message, draining rate limits and breaking prompt cache discounts.

Worse, some local tools install hidden pre-commit hooks that send your entire git diff to an LLM on every commit.

token-diet scans your repository or workstation in milliseconds to:

  • Audit prompt size: Flags instruction files that exceed context budgets ([OK], [WARN], [ALERT]).
  • Catch rogue git hooks: Detects and disables silent LLM review hooks in the active hooks directory (--fix-hooks), including linked worktrees, submodules, and repositories using core.hooksPath.
  • Locate the bloat: Breaks a Markdown directive down per level-2 heading so you know which section to trim (--sections).
  • Audit MCP overhead: Reports the context cost of every declared MCP server in opencode.json, .mcp.json, and mcp.json (--mcp).
  • Guard CI pipelines: Fails PR builds if prompt expansions exceed token thresholds or regress past a saved baseline (--strict, --max-tokens, --baseline, --max-delta).
  • Zero dependencies: Pure Python standard library (Python >= 3.8). Runs in < 10ms.

Installation

Via Homebrew (macOS & Linux)

brew install joseoper/tap/token-diet
# Or:
brew tap joseoper/tap
brew install token-diet
pipx install token-diet-cli

Via pip / uv

# Using pip:
pip install token-diet-cli

# Using uv:
uv tool install token-diet-cli

Running Directly from Source (No Installation Needed)

git clone https://github.com/joseoper/token-diet.git
cd token-diet
PYTHONPATH=src python3 -m token_diet

Usage

1. Instant Health Check in Current Directory

Simply run token-diet in any project repository. It automatically scans local directives and checks for dangerous pre-commit hooks:

token-diet

Example Output:

================================================================================
TOKEN DIET: FOOTPRINT & CONTEXT AUDIT
================================================================================
File / Directive                               |     Bytes |    Tokens |   Status
--------------------------------------------------------------------------------
/Users/developer/my-project/AGENTS.md          |       962 |       240 |     [OK]
/Users/developer/my-project/.cursorrules       |     1,240 |       310 |     [OK]
================================================================================
Total Bytes:     2,202 B
Total Tokens:    ~550 estimated base tokens per turn
Files Detected:  2 of 2

2. Automated Alerts: Context Bloat & Invisible Pre-Commit Hooks

When a prompt file is oversized or a git hook silently burns tokens on every commit, token-diet issues an instant visual alert:

token-diet

Example Alert Output:

================================================================================
TOKEN DIET: FOOTPRINT & CONTEXT AUDIT
================================================================================
File / Directive                               |     Bytes |    Tokens |   Status
--------------------------------------------------------------------------------
/Users/developer/my-project/AGENTS.md          |    24,500 |     6,125 |  [ALERT]
================================================================================
Total Bytes:     24,500 B
Total Tokens:    ~6,125 estimated base tokens per turn
Files Detected:  1 of 1

🚨 IRREGULAR HOOK DETECTED:
  • [pre-commit] Pre-commit hook invokes AI/LLM review tooling (gga, opencode). This causes invisible token consumption on every commit.
    Path: /Users/developer/my-project/.git/hooks/pre-commit
  💡 Tip: Run 'token-diet --fix-hooks' to automatically neutralize silent AI invocations.

⚠️  CONTEXT BLOAT DETECTED:
  Files marked [ALERT] exceed 16 KB (>4k tokens) and severely penalize context windows.
  Recommendation: Decouple procedures into On-Demand Skills using the LUT Router pattern.

3. Auto-Fix Silent Pre-Commit Hooks (--fix-hooks)

Neutralize unmonitored AI review tooling in .git/hooks/pre-commit in a single command with an automatic backup (.git/hooks/pre-commit.bak):

token-diet --fix-hooks

Example Output:

🛠️  AUTO-REMEDIATION: PRE-COMMIT HOOKS NEUTRALIZED
  • Backup saved: /Users/developer/my-project/.git/hooks/pre-commit.bak
  • Neutralized 1 AI review invocation(s).
  • Safe pre-commit hook is active without silent token bleed.

4. Audit System-Wide Global Prompts (--global)

Scan default assistant instruction files across your entire workstation (~/.claude/CLAUDE.md, ~/.codex/AGENTS.md, ~/.gemini/GEMINI.md, etc.):

token-diet --global

5. Audit Specific Paths or Folders

# Audit a single file
token-diet AGENTS.md

# Audit entire directories recursively
token-diet docs/

6. Automated CI / Pre-Commit Gate

Integrate into your CI pipeline or Git pre-commit workflow to prevent prompt regressions:

# Fail CI if prompt footprint exceeds 2,500 tokens:
token-diet --max-tokens 2500 AGENTS.md

# Fail CI if any file triggers a [WARN] or [ALERT]:
token-diet --strict docs/

7. Structured Telemetry Output (--json)

Emit structured JSON for automated pipelines, threshold assertions, or telemetry dashboards:

token-diet --json

8. Locate Prompt Bloat per Heading (--sections)

A monolithic directive rarely bloats evenly. This breakdown attributes bytes and estimated tokens to every level-2 (##) heading, so you know exactly which section to compress or move into an on-demand skill:

token-diet --sections AGENTS.md
SECTION BREAKDOWN: MARKDOWN HEADING FOOTPRINT
================================================================================
AGENTS.md  (23,914 B / ~5,978 tokens)
  Line | Heading                                  |     Bytes |    Tokens |   Status
--------------------------------------------------------------------------------
     1 | (preamble)                               |       412 |       103 |              [OK]
    12 | Core Directives                          |    19,220 |     4,805 |           [ALERT]
    98 | Additional Protocols                      |     4,282 |     1,070 |            [WARN]
--------------------------------------------------------------------------------

Content before the first ## heading is grouped as (preamble), and deeper headings (###) stay attributed to their parent section. Combine with --json for machine-readable output.

9. Audit MCP Server Overhead (--mcp)

Every declared MCP server consumes context, and some configs inline large tool schemas. token-diet discovers local opencode.json, .mcp.json, mcp.json, .cursor/mcp.json, and .vscode/mcp.json files and reports the payload cost per server:

token-diet --mcp
MCP SERVER PAYLOAD AUDIT
================================================================================
opencode.json  [opencode]  (2 server(s))
Server                         | Transport | Tools |     Bytes |    Tokens |   Status
--------------------------------------------------------------------------------
engram                         | local     |    12 |    18,430 |     4,607 |           [ALERT]
web-search                     | remote    |     0 |       142 |        35 |              [OK]
--------------------------------------------------------------------------------

Config-declared tools blocks are counted and included in each server's payload. Schemas fetched by a server at runtime are not visible to a local audit; the reported figure is the local injection cost floor.

10. Fail on Regression vs a Saved Baseline (--baseline, --max-delta)

Record a baseline once, then gate every pull request against it. This catches slow, incremental growth that a fixed threshold never notices:

# Save the baseline on your default branch
token-diet --json > token-baseline.json

# Fail if the total grows more than 5% in a pull request
token-diet --baseline token-baseline.json --max-delta 5

--baseline accepts any --json report ({"summary": {"total_tokens": N}}) or a minimal {"total_tokens": N} file. --max-delta exits with code 1 only when tokens increase beyond the given percentage; a decrease, an unchanged total, or growth exactly at the budget passes.


Health Status Indicators

Indicator Byte Threshold Estimated Tokens What it Means & Recommended Action
🟢 [OK] < 4 KB < 1,000 tokens Optimal Context: Fits cleanly within prompt cache. Preserves fast, cheap inference.
🟡 [WARN] 4 KB – 16 KB 1,000 – 4,000 tokens Moderate Size: Acceptable, but review for duplicate instructions or unneeded verbosity.
🔴 [ALERT] > 16 KB > 4,000 tokens Severe Prompt Bloat: Monolithic rules degrade reasoning and waste context budget every turn. Decouple into On-Demand Skills.
🚨 [HOOK ALERT] — — Hidden Token Bleed: The active pre-commit hook sends code to cloud LLMs on every git commit. Replace with static linters.

What To Do When a Leak is Detected (Action Paths)

When token-diet flags an irregularity, follow the corresponding action path (detailed runbooks available in the Remediation Playbook):

Path A: When an Irregular Git Hook is Flagged (🚨 IRREGULAR HOOK DETECTED)

  • Problem: A local hook inside .git/hooks/pre-commit secretly sends git diffs to cloud AI tools on every single commit.
  • Action Path:
    1. Inspect the script: cat .git/hooks/pre-commit
    2. Remove or comment out the automated AI review invocation.
    3. Replace it with fast, local deterministic linters (ruff, eslint, biome).
    4. Relocate semantic AI reviews to Pull Request gates in CI (see CI/CD Integration), where reviews are metered and visible.

Path B: When an Agent Directive is Bloated (🔴 [ALERT])

  • Problem: A root instruction file (AGENTS.md, CLAUDE.md, .cursorrules) exceeds 16 KB (> 4,000 tokens), exhausting your token budget on every turn and destroying cache discounts.
  • Action Path (The Look-Up Table / LUT Router Pattern):
    1. Extract Specialized Protocols: Move detailed procedure manuals, test specifications, and review arbitrations into standalone on-demand skills (e.g., skills/<skill-name>/SKILL.md).
    2. Compress Root Directive: Keep only core behavioral invariants and a routing index in the root file (< 3 KB / < 750 tokens):
      ## On-Demand Skills
      - When running database migrations, read `skills/db-migrate/SKILL.md`.
      - When preparing a release, read `skills/oss-publisher/SKILL.md`.
      
    3. Re-run token-diet to confirm the status turns 🟢 [OK].

Path C: When CI Fails on Token Budget Exceeded (exit code 1)

  • Problem: A Pull Request introduced unbudgeted prompt expansion that violated --max-tokens <N> or --strict.
  • Action Path:
    1. Run token-diet --json to get the machine-readable breakdown of offending files.
    2. Prune duplicate rules or extract text into skills.
    3. Re-run local check before pushing: token-diet --strict.

Agentic Remediation: Prompt for AI Assistants

To have your AI coding assistant (Claude Code, Gemini CLI, Cursor, Windsurf, OpenCode) remediate prompt bloat autonomously without manual edits, copy and paste this prompt:

Run `token-diet` in this repository. If you detect any files marked [ALERT] or [WARN], follow `docs/remediation-playbook.md` to decouple operational procedures into on-demand skills under `skills/<skill-name>/SKILL.md`. Compress the root instruction file to a concise Look-Up Table (LUT Router) under 3 KB (< 750 tokens). If an irregular hook is flagged, run `token-diet --fix-hooks`. Verify that all tests pass and `token-diet` reports [OK].

Running Tests

Token Diet includes an automated test suite with deterministic coverage:

python3 -m unittest discover -s tests -p 'test_*.py'

Repository Structure

token-diet/
├── .github/
│   └── workflows/
│       └── ci.yml             # Automated multi-version CI test runner
├── docs/
│   ├── remediation-playbook.md # Step-by-step token containment and LUT routing
│   └── ci-integration.md       # CI workflows & pre-commit gatekeeper examples
├── src/
│   └── token_diet/            # Core Python package (Zero external dependencies)
│       ├── __init__.py
│       ├── __main__.py
│       ├── baseline.py        # Baseline comparison & regression budget gate
│       ├── cli.py             # CLI implementation and footprint engine
│       ├── git_hooks.py       # Worktree, gitdir and core.hooksPath resolution
│       ├── mcp_audit.py       # MCP config discovery & payload accounting
│       ├── metrics.py         # Shared byte thresholds and token estimation
│       └── sections.py        # Per-heading Markdown footprint breakdown
├── tests/
│   ├── test_baseline.py       # Baseline budget unit tests
│   ├── test_git_hooks.py      # Hook resolution unit tests
│   ├── test_mcp_audit.py      # MCP payload unit tests
│   ├── test_sections.py       # Section breakdown unit tests
│   └── test_token_audit.py    # Core engine and CLI integration tests
├── .gitignore
├── AGENTS.md                  # Agent-friendly repository directives
├── CONTRIBUTING.md            # Contribution guidelines & engineering principles
├── LICENSE                    # MIT Open Source License
├── pyproject.toml             # PEP 621 packaging and CLI entrypoint
└── README.md                  # Project overview and documentation

Core Methodology: The 6 Vectors of Token Leakage

Token Diet is built on empirical analysis of how developer environments silently deplete API quotas:

  1. Global Prompt Bloat: Monolithic instruction files (CLAUDE.md, AGENTS.md) exceeding 20 KB inject tens of thousands of static tokens on every turn. Solution: Keep root directives under 3 KB (<750 tokens) and route to on-demand skills.
  2. Invisible Git Hooks: Pre-commit hooks that intercept local commits to invoke cloud LLM reviews drain quotas without developer consent. Solution: Keep local commits deterministic; reserve AI reviews for PR gates.
  3. Reasoning Effort Inflation: Running frontier models at high reasoning effort for simple syntax or diagnostic queries generates thousands of hidden chain-of-thought tokens. Solution: Default to medium or low.
  4. Static MCP Schemas: Registering dozens of MCP tools serializes massive JSON parameter schemas into system prompts. Solution: Prune unused tool servers and enforce lazy-loading.
  5. Prompt Cache Invalidation: Injecting dynamic timestamps or changing session IDs at the start of instructions destroys provider prompt caching discounts (up to 90%). Solution: Maintain immutable instruction prefixes.
  6. In-Session Runtime Inefficiencies: Agents loading entire 1,500-line files to check one function, overwriting whole files instead of focused diffs, or running uncompacted 100k+ token threads. Solution: Enforce active runtime hygiene (grep-first, line-range reads, targeted diffs, and context compaction thresholds; see Remediation Playbook Phase 4).

Platform Compatibility

Token Diet is OS-agnostic and cross-platform:

  • CLI Diagnostic Tool: Pure Python standard library with zero third-party dependencies. Runs seamlessly on macOS, Linux, and Windows (Python >= 3.8).
  • Universal Methodology: Core principles apply across all assistants (Claude Code, Gemini CLI, OpenAI Codex, OpenCode, Cursor, Windsurf).

Documentation & Guides

For in-depth operational procedures and automated pipeline setups:


License

This project is licensed under the terms of the MIT License.

Metadata

Release files for token-diet-cli 0.2.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for token-diet-cli 0.2.0
File Size Uploaded Provenance Actions
token_diet_cli-0.2.0.tar.gz 36.0 kB Without provenance Details for token_diet_cli-0.2.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for token-diet-cli 0.2.0
File Interpreter ABI Platform Actions
token_diet_cli-0.2.0-py3-none-any.whl Python 3 none any Details for token_diet_cli-0.2.0-py3-none-any.whl

Total release size: 59.2 kB

Release files / token_diet_cli-0.2.0.tar.gz

Download URL token_diet_cli-0.2.0.tar.gz
Size 36.0 kB
Tags Source
SHA-256 checksum
How to use checksums
646a2e80ae8ef24b068148056d35c34046280f1678c44caa9c04e71f27edc675
BLAKE2b-256 checksum
How to use checksums
9c406be7f82e65f721f1034d4a6029ac21c898a0bf5c67137a17bec0fcf970b8
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Release files / token_diet_cli-0.2.0-py3-none-any.whl

Download URL token_diet_cli-0.2.0-py3-none-any.whl
Size 23.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
ea2564939446b486c6856141247675c64648844b3f33b14e317e1e284e6b3688
BLAKE2b-256 checksum
How to use checksums
570d00f8ab2a4c9e0cce77565ee66496b511c56753e9440e2b8364aaf124311b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via uv/0.13.7 {"installer":{"name":"uv","version":"0.13.7","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}

Security for token-diet-cli 0.2.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page