Last released Feb 26, 2024
Minimal, fast, multi-threaded implementation of the Byte Pair Encoding (BPE) for LLM tokenization
Supported by