Last released Jun 30, 2025
High-Performance Implementation of OpenAI's TikToken - 2x Throughput, 4x Faster Code Tokenization
Supported by