tiktoken-tools
A set of command-line tools for estimating how many tokens a file will consume when sent to an LLM API. Useful for cost estimation, context window planning, and pre-processing workflows.
Why this exists
The primary use case is token-free assessment of document token costs - whether you're checking a single file or scanning tens of thousands. This makes it practical to build pre-processing pipelines that gate on token budgets before paying API costs, without making any API calls at all.
I use tiktoken-tools extensively in my own Claude Code workflow. The included skills are integrated into many of my personal automation scripts to drive efficient token usage at scale - flagging large files before they're read into context, estimating API costs before batch jobs, and keeping sessions within context-window limits.
Supports text files, PDFs (both text-extractable and scanned), images, Excel spreadsheets, CSVs, and Word documents (including embedded images). Text tokenization uses OpenAI's tiktoken library with the o200k_base encoding (GPT-4o family). Image token costs are calculated using the published formulas for OpenAI and Anthropic APIs.
Supported file types
| Type | Extensions | What it counts |
|---|---|---|
| Text | .txt, .md, .py, .js, etc. | tiktoken tokens |
| PDF (text) | tiktoken tokens from extracted text | |
| PDF (scanned) | Image tokens per rendered page | |
| Image | .png, .jpg, .gif, .webp, etc. | OpenAI/Anthropic image tokens |
| XLSX | .xlsx | tiktoken tokens from CSV text representation |
| CSV | .csv | tiktoken tokens + row/col stats |
| DOCX | .docx | tiktoken tokens (text) + image tokens (embedded images) |
Installation
Full install with Claude Code integration (recommended)
Clone the repo and run the installer. This installs the CLI tools, symlinks the Claude Code skills into ~/.claude/skills/, and adds a token-awareness section to ~/.claude/CLAUDE.md:
git clone https://github.com/raffishquartan/tiktoken-tools.git
cd tiktoken-tools
./install-everything.sh
Requires uv.
CLI only
If you only want the command-line tools without the Claude Code skills, install from PyPI:
uv tool install tiktoken-tools
# or
pip install tiktoken-tools
To install the latest unreleased code straight from GitHub instead:
uv tool install git+https://github.com/raffishquartan/tiktoken-tools.git
Usage
Generic dispatcher (auto-detects file type)
count-tokens -f document.pdf
count-tokens -f photo.png
count-tokens -f data.xlsx
# Directory - recursively counts all files and outputs CSV to stdout
count-tokens -f ./my-project/
count-tokens -f ./my-project/ > token-counts.csv
# Stdin
echo "hello world" | count-tokens -f -
Type-specific commands
# Text files (supports stdin via -f -)
count-text-tokens -f readme.md
echo "hello world" | count-text-tokens -f -
# PDF (text-based)
count-pdf-tokens -f document.pdf --csv page-stats.csv
# Images
count-image-tokens -f photo.png
count-image-tokens -f photo.png --provider anthropic
count-image-tokens -f photo.png --detail low
# Scanned/image PDFs
count-image-pdf-tokens -f scanned.pdf --provider openai --dpi 150
# Excel
count-xlsx-tokens -f spreadsheet.xlsx
# CSV
count-csv-tokens -f data.csv
# Word documents (text + embedded images)
count-docx-tokens -f report.docx --provider anthropic
Common flags
| Flag | Description | Applies to |
|---|---|---|
-f, --file |
Path to file, directory, or - for stdin |
All |
-m, --model |
Model for tiktoken encoding (default: gpt-4o-mini) | Text-based types |
-p, --provider |
openai or anthropic (default: highest count) |
Image-based types |
-d, --detail |
OpenAI detail level: low or high (default: high) |
Image-based types |
Image token formulas
OpenAI (GPT-4o / GPT-4o-mini) - high detail
- Scale longest side to <= 2048px (downscale only)
- Scale shortest side to <= 768px (downscale only)
- Tile into 512x512 blocks
tokens = tiles * 170 + 85
Low detail: flat 85 tokens.
Anthropic (Claude 3+)
- Scale so neither dimension exceeds 1568px (downscale only)
tokens = ceil(width * height / 750)
A note on Anthropic token counts
Anthropic's Claude models use a custom BPE tokenizer that is not publicly documented. tiktoken uses OpenAI's o200k_base encoding, which differs from Claude's tokenizer. In practice the counts are directionally accurate - differences for English prose are typically small (within a few percent) - but can be more pronounced for code, non-English text, or content heavy with special characters.
Treat tiktoken counts as planning estimates ("will this fit in my context window?", "roughly how much will this batch cost?") rather than exact billing figures. The Anthropic API returns authoritative input_tokens counts in every response; use those for precise accounting.
PDF classification
The generic count-tokens command auto-detects whether a PDF contains extractable text or is scanned/image-based. If the average extracted text per page is less than 50 characters, it's treated as an image PDF and pages are rendered at 150 DPI for token counting.
Development
To fork, develop, or contribute:
git clone https://github.com/raffishquartan/tiktoken-tools.git
cd tiktoken-tools
uv pip install -e ".[dev]"
pytest
Claude Code integration
tiktoken-tools includes four Claude Code skills that let Claude proactively monitor and manage token usage during sessions:
| Skill | Purpose |
|---|---|
count-file-tokens |
Count tokens in any file before reading it into context |
estimate-session-tokens |
Estimate how many tokens the current session has consumed |
estimate-api-cost |
Estimate the cost of API calls given a token count |
check-token-fit-for-session |
Check whether a set of files will fit in the remaining context |
The easiest way to install them is ./install-everything.sh (see Installation above). The skills are symlinked, so pulling repo updates automatically updates the installed skills.
To configure Claude to use the skills proactively, add this to your ~/.claude/CLAUDE.md (the installer does this for you):
# Token awareness
Use the tiktoken-tools skills (count-file-tokens, estimate-session-tokens, estimate-api-cost,
check-token-fit-for-session) proactively to monitor and manage token usage. Before reading large
files, check their token count with /count-file-tokens. When the session has been running for a
while, use /estimate-session-tokens to check context usage and warn if approaching limits. When
the user is planning API calls or batch processing, use /estimate-api-cost and
/check-token-fit-for-session to help them understand costs and capacity. Always present token
counts concisely and suggest ways to reduce token usage when files are large (e.g. converting
xlsx to CSV, using low-detail mode for images, extracting relevant pages from PDFs).
For full skill installation details, see claude-skills/README.md.
Dependencies
Metadata
Release files for tiktoken-tools 0.1.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| tiktoken_tools-0.1.0.tar.gz | 22.9 kB | With provenance | Details for tiktoken_tools-0.1.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| tiktoken_tools-0.1.0-py3-none-any.whl | Python 3 | none | any | Details for tiktoken_tools-0.1.0-py3-none-any.whl |
Total release size: 40.9 kB
Release files / tiktoken_tools-0.1.0.tar.gz
| Download URL | tiktoken_tools-0.1.0.tar.gz |
|---|---|
| Size | 22.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
02b3405bf07ba09a4df7799ba73fbf30749bc84d9e85a374629041db33646846
|
|
BLAKE2b-256 checksum How to use checksums |
0143d3c31b9311e981506bb2bc2a0394d96755287e50bc76d03903e8197ac704
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 11, 2026.
Transparency logRelease files / tiktoken_tools-0.1.0-py3-none-any.whl
| Download URL | tiktoken_tools-0.1.0-py3-none-any.whl |
|---|---|
| Size | 18.0 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
34916fb8dc62d201c0647ca7642f953b30279b3bb0544b6391e03f0d8be6e0ba
|
|
BLAKE2b-256 checksum How to use checksums |
1046685e62ff16c39061da60fb7c0eb816e01f2f808e3a2187eea46bbbbeb70b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
twine/6.1.0 CPython/3.13.12
|
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 11, 2026.
Transparency logSecurity for tiktoken-tools 0.1.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
Full provenance
Both files in this release have been uploaded with provenance metadata.
Attestation for tiktoken_tools-0.1.0.tar.gz
PyPI Publish Attestation
PyPI verified that this artifact, at this checksum, originated from the publisher listed below.
Signed by GitHub Actions, verified by PyPI on Jul 11, 2026.
Transparency logA note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.