Last released Dec 20, 2024
Fast and versatile tokenizer for language models, supporting BPE, Unigram and WordPiece tokenization
Supported by
