Last released Oct 26, 2026
Fast tokenizer for language models, supporting BPE, Unigram and WordPiece tokenization