kitoken 0.10.0

Fast and versatile tokenizer for language models, supporting BPE, Unigram and WordPiece tokenization

kitoken

Tokenizer for language models.

from kitoken import Kitoken

const encoder = Kitoken.from_file("models/llama2.kit")

const tokens = encoder.encode("hello world!", True)
const string = encoder.decode(tokens).decode("utf-8")

assert string == "hello world!"

Features

• Fast encoding and decoding
  Faster than most other tokenizers in both common and uncommon scenarios.
• Support for a wide variety of tokenizer formats and tokenization strategies
  Including support for Tokenizers, SentencePiece, Tiktoken and more.
• Compatible with many systems and platforms
  Runs on Windows, Linux, macOS and embedded, and comes with bindings for Web, Node and Python.
• Compact data format
  Definitions are stored in an efficient binary format and without merge list.
• Support for normalization and pre-tokenization
  Including unicode normalization, whitespace normalization, and many others.

Overview

Kitoken is a fast and versatile tokenizer for language models. Multiple tokenization algorithms are supported:

• BytePair: A variation of the BPE algorithm, merging byte or character pairs.
• Unigram: The Unigram subword algorithm.
• WordPiece: The WordPiece subword algorithm.

Kitoken is compatible with many existing tokenizers, including SentencePiece, HuggingFace Tokenizers, OpenAI Tiktoken and Mistral Tekken.

See the main README for more information.