count-tokens-cli 0.1.0

Count tokens in git-tracked files using tiktoken encoding for LLM API cost estimation

Count Tokens

A fast CLI tool that counts tokens in files using tiktoken encoding, automatically respecting .gitignore patterns. Perfect for estimating LLM API costs before processing your codebase.

Why Count Tokens?

When working with LLMs, understanding token usage helps you:

• Estimate API costs before sending large codebases to models
• Stay within context limits for different LLM models
• Optimize prompts by identifying the largest files
• Budget effectively for AI-assisted development workflows

Installation

From PyPI

pip install count-tokens-cli

From Source

git clone https://github.com/StGerman/count-tokens.git
cd count-tokens
poetry install

Quick Start

# Count tokens in your current repository
count-tokens

# Get just the summary without file breakdown
count-tokens --no-files

# Focus on specific file types
count-tokens --extensions .py .js .ts .md

Usage Examples

Basic Token Counting

count-tokens

Top 4 files by token count:
============================================================
   1,209 tokens | src/main.py
     639 tokens | docs/api.md
     290 tokens | README.md
     138 tokens | config.yaml
============================================================

Total tokens: 2,276
Total files: 4
Average per file: 569 tokens

Cost estimates (per API call):
  Claude Sonnet 4    $0.0068
  Claude Opus 4      $0.0341
  GPT-4o             $0.0057
  GPT-4o mini        $0.0003
  GPT-4 Turbo        $0.0228

Summary Only

count-tokens --no-files

Perfect for CI/CD pipelines or when you just need the totals.

Custom File Types

count-tokens --extensions .py .js .md

Focus on specific languages or documentation files.

Different Encodings

count-tokens --encoding o200k_base

Use different tiktoken encodings (cl100k_base, o200k_base, etc.).

Show More Files

count-tokens --top 50

Display up to 50 files instead of the default 30.

Supported File Types

The tool automatically detects and counts tokens in 25+ common file extensions:

Languages: .py .js .ts .jsx .tsx .java .go .rb .php .c .cpp .h .cs .swift .kt .rs .scala .r .m .sh

Config & Data: .yml .yaml .json .xml .toml .sql

Web & Docs: .html .css .scss .vue .md

Features

✅ Gitignore Aware - Automatically respects .gitignore patterns
✅ No Git Required - Works in any directory, not just git repositories
✅ Smart Filtering - Supports 25+ code file extensions
✅ Cost Estimation - Real-time pricing for popular LLM APIs
✅ Flexible Encodings - Works with all tiktoken encodings
✅ Top Files View - Identify your largest files quickly
✅ Error Resilient - Gracefully handles unreadable files
✅ Fast Performance - Optimized for large repositories

Development

Setup

# Install dependencies
poetry install

# Run during development
poetry run count-tokens
# or make executable and run directly
chmod +x count_tokens.py
./count_tokens.py

Building & Publishing

# Build package
poetry build

# Publish to PyPI
poetry publish

Testing

# Test on current repository
poetry run count-tokens

# Test with different options
poetry run count-tokens --no-files --encoding o200k_base

Requirements

• Python 3.11+
• tiktoken library for token encoding
• pathspec library for .gitignore pattern matching

How It Works

The tool walks through all files in the current directory and subdirectories, automatically excluding files that match patterns in:

• .gitignore files (current directory and parent directories)
• Common ignore patterns (.git/ directories, etc.)

This means you get clean token counts without build artifacts, dependencies, or other files you typically don't want to include in LLM contexts.

Contributing

Contributions welcome! Please feel free to submit issues or pull requests.

License

MIT License - see LICENSE file for details.