repotoken 0.1.3

Count tokens across any code repository — fast, local, no API calls

repotoken

Count tokens across any code repository — fast, local, no API calls.

$ repotoken .

scanning /your/project ...

  file                      tokens              lines
  ────────────────────────  ────────         ──────
  src/parser.py               3.1K              140
  src/main.py                 2.8K              120
  src/utils.py                1.2K               55
  tests/test_parser.py         890               40
  README.md                    430               30

  ──────────────────────────────────────────────────
  TOTAL                       8.4K tokens   385 lines
  files                                           5
  skipped (binary)                                3
  ──────────────────────────────────────────────────

Why?

LLMs have context limits — GPT-4o tops out at 128K tokens, most Claude models at 200K. If you're working with AI tooling on a codebase, you need to know how big your repo actually is in token terms — not lines, not bytes, but tokens.

repotoken tells you exactly that, per file and in total, so you can make informed decisions about chunking, batching, or summarisation.

Install

pip install repotoken

Usage

# scan current directory
repotoken .

# scan any repo
repotoken /path/to/any/repo

Notes

• Uses cl100k_base encoding (GPT-4 / GPT-4o tokenizer) — accurate for most modern LLMs within ~10-15%
• Detects binaries using null-byte scanning (same method as git) — no hardcoded extension lists
• Automatically skips .git, node_modules, __pycache__, .venv, and other noise
• Sorts output by token count so the heaviest files are at the top
• No API calls, no network requests, runs entirely locally

License

MIT