tiktoken-tools 0.1.0

Token counting tools for text, PDF, images, xlsx, csv, and docx files

tiktoken-tools

A set of command-line tools for estimating how many tokens a file will consume when sent to an LLM API. Useful for cost estimation, context window planning, and pre-processing workflows.

Why this exists

The primary use case is token-free assessment of document token costs - whether you're checking a single file or scanning tens of thousands. This makes it practical to build pre-processing pipelines that gate on token budgets before paying API costs, without making any API calls at all.

I use tiktoken-tools extensively in my own Claude Code workflow. The included skills are integrated into many of my personal automation scripts to drive efficient token usage at scale - flagging large files before they're read into context, estimating API costs before batch jobs, and keeping sessions within context-window limits.

Supports text files, PDFs (both text-extractable and scanned), images, Excel spreadsheets, CSVs, and Word documents (including embedded images). Text tokenization uses OpenAI's tiktoken library with the o200k_base encoding (GPT-4o family). Image token costs are calculated using the published formulas for OpenAI and Anthropic APIs.

Supported file types

Type	Extensions	What it counts
Text	.txt, .md, .py, .js, etc.	tiktoken tokens
PDF (text)	.pdf	tiktoken tokens from extracted text
PDF (scanned)	.pdf	Image tokens per rendered page
Image	.png, .jpg, .gif, .webp, etc.	OpenAI/Anthropic image tokens
XLSX	.xlsx	tiktoken tokens from CSV text representation
CSV	.csv	tiktoken tokens + row/col stats
DOCX	.docx	tiktoken tokens (text) + image tokens (embedded images)

Installation

Full install with Claude Code integration (recommended)

Clone the repo and run the installer. This installs the CLI tools, symlinks the Claude Code skills into ~/.claude/skills/, and adds a token-awareness section to ~/.claude/CLAUDE.md:

git clone https://github.com/raffishquartan/tiktoken-tools.git
cd tiktoken-tools
./install-everything.sh

Requires uv.

CLI only

If you only want the command-line tools without the Claude Code skills, install from PyPI:

uv tool install tiktoken-tools
# or
pip install tiktoken-tools

To install the latest unreleased code straight from GitHub instead:

uv tool install git+https://github.com/raffishquartan/tiktoken-tools.git

Usage

Generic dispatcher (auto-detects file type)

count-tokens -f document.pdf
count-tokens -f photo.png
count-tokens -f data.xlsx

# Directory - recursively counts all files and outputs CSV to stdout
count-tokens -f ./my-project/
count-tokens -f ./my-project/ > token-counts.csv

# Stdin
echo "hello world" | count-tokens -f -

Type-specific commands

# Text files (supports stdin via -f -)
count-text-tokens -f readme.md
echo "hello world" | count-text-tokens -f -

# PDF (text-based)
count-pdf-tokens -f document.pdf --csv page-stats.csv

# Images
count-image-tokens -f photo.png
count-image-tokens -f photo.png --provider anthropic
count-image-tokens -f photo.png --detail low

# Scanned/image PDFs
count-image-pdf-tokens -f scanned.pdf --provider openai --dpi 150

# Excel
count-xlsx-tokens -f spreadsheet.xlsx

# CSV
count-csv-tokens -f data.csv

# Word documents (text + embedded images)
count-docx-tokens -f report.docx --provider anthropic

Common flags

Flag	Description	Applies to
-f, --file	Path to file, directory, or - for stdin	All
-m, --model	Model for tiktoken encoding (default: gpt-4o-mini)	Text-based types
-p, --provider	openai or anthropic (default: highest count)	Image-based types
-d, --detail	OpenAI detail level: low or high (default: high)	Image-based types

Image token formulas

OpenAI (GPT-4o / GPT-4o-mini) - high detail

1. Scale longest side to <= 2048px (downscale only)
2. Scale shortest side to <= 768px (downscale only)
3. Tile into 512x512 blocks
4. tokens = tiles * 170 + 85

Low detail: flat 85 tokens.

Anthropic (Claude 3+)

1. Scale so neither dimension exceeds 1568px (downscale only)
2. tokens = ceil(width * height / 750)

A note on Anthropic token counts

Anthropic's Claude models use a custom BPE tokenizer that is not publicly documented. tiktoken uses OpenAI's o200k_base encoding, which differs from Claude's tokenizer. In practice the counts are directionally accurate - differences for English prose are typically small (within a few percent) - but can be more pronounced for code, non-English text, or content heavy with special characters.

Treat tiktoken counts as planning estimates ("will this fit in my context window?", "roughly how much will this batch cost?") rather than exact billing figures. The Anthropic API returns authoritative input_tokens counts in every response; use those for precise accounting.

PDF classification

The generic count-tokens command auto-detects whether a PDF contains extractable text or is scanned/image-based. If the average extracted text per page is less than 50 characters, it's treated as an image PDF and pages are rendered at 150 DPI for token counting.

Development

To fork, develop, or contribute:

git clone https://github.com/raffishquartan/tiktoken-tools.git
cd tiktoken-tools
uv pip install -e ".[dev]"
pytest

Claude Code integration

tiktoken-tools includes four Claude Code skills that let Claude proactively monitor and manage token usage during sessions:

Skill	Purpose
count-file-tokens	Count tokens in any file before reading it into context
estimate-session-tokens	Estimate how many tokens the current session has consumed
estimate-api-cost	Estimate the cost of API calls given a token count
check-token-fit-for-session	Check whether a set of files will fit in the remaining context

The easiest way to install them is ./install-everything.sh (see Installation above). The skills are symlinked, so pulling repo updates automatically updates the installed skills.

To configure Claude to use the skills proactively, add this to your ~/.claude/CLAUDE.md (the installer does this for you):

# Token awareness

Use the tiktoken-tools skills (count-file-tokens, estimate-session-tokens, estimate-api-cost,
check-token-fit-for-session) proactively to monitor and manage token usage. Before reading large
files, check their token count with /count-file-tokens. When the session has been running for a
while, use /estimate-session-tokens to check context usage and warn if approaching limits. When
the user is planning API calls or batch processing, use /estimate-api-cost and
/check-token-fit-for-session to help them understand costs and capacity. Always present token
counts concisely and suggest ways to reduce token usage when files are large (e.g. converting
xlsx to CSV, using low-detail mode for images, extracting relevant pages from PDFs).

For full skill installation details, see claude-skills/README.md.

Dependencies

• tiktoken - text tokenization
• Pillow - image dimension reading
• PyMuPDF - PDF text extraction and page rendering
• pypdf - fallback PDF text extraction
• openpyxl - xlsx reading
• python-docx - docx reading