gemini-tree-token-counter 0.1.0

Estimate tokens for Gemini and display tree from local/git sources.

Gemini Tree Token Counter (gtc)

🇯🇵 日本語 (Japanese)

A CLI tool to estimate tokens for Gemini models across local directories and GitHub repositories. It visualizes the file tree with token counts and supports "Time Travel" (analyzing specific Git commits/dates).

Note: The token counting logic is based on regex approximations (ported from smartprocure/gemini-token-estimator) and is designed for speed. It is accurate within ~10% for most cases.

Features

• 🚀 Fast Estimation: Uses regex-based logic (no API calls required).
• 🌳 Tree Visualization: Displays directory structure with token counts per node.
• 🐙 GitHub Support: Analyze remote repositories directly.
• ⏳ Time Travel: Check token counts for a specific branch, commit hash, or date (e.g., "how many tokens was this repo in 2023?").
• 📄 Content Inspection: Optionally output file contents for context generation.

Installation

Prerequisites

• Python 3.7+
• Git (The git command must be available in your system path)

Install via pip

pip install gemini-tree-token-counter

Usage

Basic usage scans the current directory:

gtc

Options

• target: Local path or GitHub URL (default: current directory).
• -d, --dir: Target specific subdirectories (can be used multiple times).
• -c, --content: Display file contents (useful for piping to LLMs).
• -b, --branch: Checkout a specific Git branch.
• --commit: Checkout a specific Git commit hash.
• --date: Checkout the latest commit before a specific date (format: YYYY-MM-DD).
• --first: Checkout the first (initial) commit of the repo.

Examples

# 1. Local Directory
gtc                       # Scan current directory
gtc . -d src -d lib       # Scan specific dirs
gtc . -c > context.txt    # Dump all code and tokens to a file

# 2. GitHub Repository
gtc https://github.com/user/repo
gtc https://github.com/user/repo -b develop

# 3. Time Travel (Git History)
gtc . --date 2023-01-01   # How big was this project last year?
gtc . --first             # How small was the first commit?

License & Acknowledgments

This project is licensed under the MIT License.

However, the tokenizer regex logic is a port of gemini-token-estimator by GovSpend, which is licensed under the ISC License.

Tokenizer Logic Copyright

Copyright (c) GovSpend

Permission to use, copy, modify, and/or distribute this software for any purpose with or without fee is hereby granted, provided that the above copyright notice and this permission notice appear in all copies.