GPT-Tokenizer

A terminal-based GPT token visualiser. Type text into an interactive REPL and each token is printed on a colored background so you can see exactly where the tokenizer splits your input. Backed by tiktoken.

Install

Requires Python 3.10+.

python -m venv .venv
source .venv/bin/activate
pip install -e .

Usage

gpt-tokenizer          # console entry point
# or
python -m gpt_tokenizer

Type text at the prompt and press Enter to tokenize it. To write multiple lines or paragraphs, press Shift+Enter (or Option/Alt+Enter) to insert a newline without submitting — the whole block is tokenized together. Press Ctrl-D to quit.

Shift+Enter relies on your terminal reporting it as a distinct key (iTerm2, kitty, WezTerm, and the VS Code terminal do; macOS Terminal.app does not). Option/Alt+Enter works everywhere.

A line on its own starting with / is a command (runs on Enter):

Newlines in your text are shown as a dim ⏎ glyph so the original layout stays visible. The default model is gpt-4o.

Project layout

src/gpt_tokenizer/     Installable package
  cli.py               REPL loop and command dispatch
  tokenizer.py         tiktoken encode wrapper -> list of (id, bytes)
  models.py            Supported models + interactive selection
  render.py            Banner and colorized token output
  ansi.py              ANSI escape-sequence helpers
examples/
  bpe_from_scratch.py  Standalone, educational BPE walkthrough (Karpathy minbpe style)
data/                  GPT-2 encoder.json / vocab.bpe (used only by the example)
tests/                 pytest suite

Development

pip install -e ".[dev]"      # installs pytest
pytest

pip install -e ".[examples]" # installs regex, needed by the BPE example
python examples/bpe_from_scratch.py