rtok

A Python tokenizer for LLMs using GitHub's linear BPE implementation.

[rtok]
ebook (90k) 0.12907366300350986
a*100 7.582298712804914e-05
a*1000 0.00013318200944922864
a*10000 0.0013674889924004674
a*100000 0.01401260400598403

[tiktoken]
ebook (90k) 0.23613008800020907
a*100 0.00018489900685381144
a*1000 0.003490732007776387
a*10000 0.3407805879978696
a*100000 33.41563105300884

API

rtok.openai.get_o200k_base() -> Encoder
rtok.openai.get_cl100k_base() -> Encoder
Encoder.count(str)
Encoder.count_till_limit(str, limit: int) -> Optional[int]
Encoder.encode(str) -> [int]
Encoder.decode([int]) -> str

Encoder.encode and Encoder.decode are compatible with tiktoken. See test.py. Encoder.count_till_limit() returns None if the count exceeds the limit.