Last released May 10, 2026
Fast tokenizer for language models, supporting BPE, Unigram and WordPiece tokenization
Supported by