repotoken 0.1.0

Count tokens across any code repository — fast, local, no API calls

repotoken

Count tokens across any code repository — fast, local, no API calls.

$ repotoken .

scanning /your/project ...

  src/parser.py          3.1K tokens
  src/main.py            2.8K tokens
  src/utils.py           1.2K tokens
  tests/test_parser.py     890 tokens
  README.md                430 tokens

  ──────────────────────────────────
  TOTAL                    8.4K tokens
  files scanned                   5
  skipped (binary)                3

Why?

LLMs have context limits — GPT-4o tops out at 128K tokens, most Claude models at 200K. If you're working with AI tooling on a codebase, you need to know how big your repo actually is in token terms — not lines, not bytes, but tokens.

repotoken tells you exactly that, per file and in total, so you can make informed decisions about chunking, batching, or summarisation.

Install

pip install repotoken

Usage

# scan current directory
repotoken .

# scan any repo
repotoken /path/to/any/repo

Notes

• Uses cl100k_base encoding (GPT-4 / GPT-4o tokenizer) — accurate for most modern LLMs within ~10-15%
• Detects binaries using null-byte scanning (same method as git) — no hardcoded extension lists
• Automatically skips .git, node_modules, __pycache__, .venv, and other noise
• Sorts output by token count so the heaviest files are at the top
• No API calls, no network requests, runs entirely locally

License

MIT