Skip to main content

tiktoken-tools

A set of command-line tools for estimating how many tokens a file will consume when sent to an LLM API. Useful for cost estimation, context window planning, and pre-processing workflows.

Why this exists

The primary use case is token-free assessment of document token costs - whether you're checking a single file or scanning tens of thousands. This makes it practical to build pre-processing pipelines that gate on token budgets before paying API costs, without making any API calls at all.

I use tiktoken-tools extensively in my own Claude Code workflow. The included skills are integrated into many of my personal automation scripts to drive efficient token usage at scale - flagging large files before they're read into context, estimating API costs before batch jobs, and keeping sessions within context-window limits.

Supports text files, PDFs (both text-extractable and scanned), images, Excel spreadsheets, CSVs, and Word documents (including embedded images). Text tokenization uses OpenAI's tiktoken library with the o200k_base encoding (GPT-4o family). Image token costs are calculated using the published formulas for OpenAI and Anthropic APIs.

Supported file types

Type Extensions What it counts
Text .txt, .md, .py, .js, etc. tiktoken tokens
PDF (text) .pdf tiktoken tokens from extracted text
PDF (scanned) .pdf Image tokens per rendered page
Image .png, .jpg, .gif, .webp, etc. OpenAI/Anthropic image tokens
XLSX .xlsx tiktoken tokens from CSV text representation
CSV .csv tiktoken tokens + row/col stats
DOCX .docx tiktoken tokens (text) + image tokens (embedded images)

Installation

Clone the repo and run the installer. This installs the CLI tools, symlinks the Claude Code skills into ~/.claude/skills/, and adds a token-awareness section to ~/.claude/CLAUDE.md:

git clone https://github.com/raffishquartan/tiktoken-tools.git
cd tiktoken-tools
./install-everything.sh

Requires uv.

CLI only

If you only want the command-line tools without the Claude Code skills, install from PyPI:

uv tool install tiktoken-tools
# or
pip install tiktoken-tools

To install the latest unreleased code straight from GitHub instead:

uv tool install git+https://github.com/raffishquartan/tiktoken-tools.git

Usage

Generic dispatcher (auto-detects file type)

count-tokens -f document.pdf
count-tokens -f photo.png
count-tokens -f data.xlsx

# Directory - recursively counts all files and outputs CSV to stdout
count-tokens -f ./my-project/
count-tokens -f ./my-project/ > token-counts.csv

# Stdin
echo "hello world" | count-tokens -f -

Type-specific commands

# Text files (supports stdin via -f -)
count-text-tokens -f readme.md
echo "hello world" | count-text-tokens -f -

# PDF (text-based)
count-pdf-tokens -f document.pdf --csv page-stats.csv

# Images
count-image-tokens -f photo.png
count-image-tokens -f photo.png --provider anthropic
count-image-tokens -f photo.png --detail low

# Scanned/image PDFs
count-image-pdf-tokens -f scanned.pdf --provider openai --dpi 150

# Excel
count-xlsx-tokens -f spreadsheet.xlsx

# CSV
count-csv-tokens -f data.csv

# Word documents (text + embedded images)
count-docx-tokens -f report.docx --provider anthropic

Common flags

Flag Description Applies to
-f, --file Path to file, directory, or - for stdin All
-m, --model Model for tiktoken encoding (default: gpt-4o-mini) Text-based types
-p, --provider openai or anthropic (default: highest count) Image-based types
-d, --detail OpenAI detail level: low or high (default: high) Image-based types

Image token formulas

OpenAI (GPT-4o / GPT-4o-mini) - high detail

  1. Scale longest side to <= 2048px (downscale only)
  2. Scale shortest side to <= 768px (downscale only)
  3. Tile into 512x512 blocks
  4. tokens = tiles * 170 + 85

Low detail: flat 85 tokens.

Anthropic (Claude 3+)

  1. Scale so neither dimension exceeds 1568px (downscale only)
  2. tokens = ceil(width * height / 750)

A note on Anthropic token counts

Anthropic's Claude models use a custom BPE tokenizer that is not publicly documented. tiktoken uses OpenAI's o200k_base encoding, which differs from Claude's tokenizer. In practice the counts are directionally accurate - differences for English prose are typically small (within a few percent) - but can be more pronounced for code, non-English text, or content heavy with special characters.

Treat tiktoken counts as planning estimates ("will this fit in my context window?", "roughly how much will this batch cost?") rather than exact billing figures. The Anthropic API returns authoritative input_tokens counts in every response; use those for precise accounting.

PDF classification

The generic count-tokens command auto-detects whether a PDF contains extractable text or is scanned/image-based. If the average extracted text per page is less than 50 characters, it's treated as an image PDF and pages are rendered at 150 DPI for token counting.

Development

To fork, develop, or contribute:

git clone https://github.com/raffishquartan/tiktoken-tools.git
cd tiktoken-tools
uv pip install -e ".[dev]"
pytest

Claude Code integration

tiktoken-tools includes four Claude Code skills that let Claude proactively monitor and manage token usage during sessions:

Skill Purpose
count-file-tokens Count tokens in any file before reading it into context
estimate-session-tokens Estimate how many tokens the current session has consumed
estimate-api-cost Estimate the cost of API calls given a token count
check-token-fit-for-session Check whether a set of files will fit in the remaining context

The easiest way to install them is ./install-everything.sh (see Installation above). The skills are symlinked, so pulling repo updates automatically updates the installed skills.

To configure Claude to use the skills proactively, add this to your ~/.claude/CLAUDE.md (the installer does this for you):

# Token awareness

Use the tiktoken-tools skills (count-file-tokens, estimate-session-tokens, estimate-api-cost,
check-token-fit-for-session) proactively to monitor and manage token usage. Before reading large
files, check their token count with /count-file-tokens. When the session has been running for a
while, use /estimate-session-tokens to check context usage and warn if approaching limits. When
the user is planning API calls or batch processing, use /estimate-api-cost and
/check-token-fit-for-session to help them understand costs and capacity. Always present token
counts concisely and suggest ways to reduce token usage when files are large (e.g. converting
xlsx to CSV, using low-detail mode for images, extracting relevant pages from PDFs).

For full skill installation details, see claude-skills/README.md.

Dependencies

Metadata

Release files for tiktoken-tools 0.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for tiktoken-tools 0.1.0
File Size Uploaded Provenance Actions
tiktoken_tools-0.1.0.tar.gz 22.9 kB With provenance Details for tiktoken_tools-0.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for tiktoken-tools 0.1.0
File Interpreter ABI Platform Actions
tiktoken_tools-0.1.0-py3-none-any.whl Python 3 none any Details for tiktoken_tools-0.1.0-py3-none-any.whl

Total release size: 40.9 kB

Release files / tiktoken_tools-0.1.0.tar.gz

Download URL tiktoken_tools-0.1.0.tar.gz
Size 22.9 kB
Tags Source
SHA-256 checksum
How to use checksums
02b3405bf07ba09a4df7799ba73fbf30749bc84d9e85a374629041db33646846
BLAKE2b-256 checksum
How to use checksums
0143d3c31b9311e981506bb2bc2a0394d96755287e50bc76d03903e8197ac704
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.12

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Jul 11, 2026.

Transparency log

Release files / tiktoken_tools-0.1.0-py3-none-any.whl

Download URL tiktoken_tools-0.1.0-py3-none-any.whl
Size 18.0 kB
Tags Python 3
SHA-256 checksum
How to use checksums
34916fb8dc62d201c0647ca7642f953b30279b3bb0544b6391e03f0d8be6e0ba
BLAKE2b-256 checksum
How to use checksums
1046685e62ff16c39061da60fb7c0eb816e01f2f808e3a2187eea46bbbbeb70b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/6.1.0 CPython/3.13.12

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Jul 11, 2026.

Transparency log

Security for tiktoken-tools 0.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for tiktoken_tools-0.1.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Jul 11, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.0 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page