bpe-from-scratch 0.2.0

Byte-pair encoding tokenizer built from scratch

BPE from Scratch

A ground-up Python implementation of Byte-Level Byte Pair Encoding (BBPE) — the tokenization algorithm used by GPT-2, GPT-3, GPT-4, and other modern LLMs.

What is Byte-Level BPE?

BPE was originally a data compression algorithm that replaces the most frequent pair of bytes in a sequence with a single unused symbol. Applied to NLP, it builds a subword vocabulary by iteratively merging the most frequent adjacent token pairs in a corpus.

The byte-level variant (introduced by OpenAI for GPT-2) operates directly on raw UTF-8 bytes rather than characters or words:

• Base vocabulary of 256 — one token per possible byte value (0–255), no unknown tokens ever
• Language-agnostic — any Unicode text (code, math, emoji, CJK, ...) is representable without a special <UNK> token
• Lossless — encoding and decoding are exact roundtrips
• Merges learned greedily — at each step, the most frequent adjacent pair is merged and assigned a new token ID (256, 257, ...)

This is the same fundamental approach used by tiktoken (OpenAI) and Hugging Face tokenizers for GPT-style models.

Algorithm Phases

Phase	Description
1	UTF-8 encoding — normalize and encode input text to bytes
2	Byte → token conversion — represent each byte as an integer token ID in [0, 255]
3	Pair counting — count all adjacent token pairs in the sequence
4	Merge — replace the most frequent pair with a new token ID
5	Repeat — iterate until the target vocabulary size is reached

Project Structure

src/bpe.py        # Core implementation
tests/test_bpe.py # Unit tests
tests/manual/     # Interactive notebooks for experimentation

Running Tests

PYTHONPATH=src python3 -m unittest discover -s tests -v

Acknowledgements

Inspired by Andrej Karpathy's minbpe.

References

• minbpe — Minimal BPE implementation by Andrej Karpathy
• GPT-2 Paper — Language Models are Unsupervised Multitask Learners (Radford et al., 2019)
• Byte-Pair Encoding tokenization — Hugging Face NLP Course
• Neural Machine Translation of Rare Words with Subword Units — original BPE-for-NLP paper (Sennrich et al., 2016)