Skip to main content

No project description provided

Project description

🪙 toktokenizer

toktokenizer is a BPE tokenizer implemented in rust and exposed in python using pyo3 bindings.

import toktokenizer as tok
bpe = tok.BPETokenizer.from_pretrained("wikibpe.json")

assert bpe.decode(bpe.encode("rust is pretty fun 🦀"))

Install toktokenizer from PyPI with the following

pip install toktokenizer

Note: if you want to build from source make sure rust is installed!

The only class toktokenizer exposes is BPETokenizer. The class itself is pretty minimalistic, with all major methods being showed below:

from toktokenizer import BPETokenizer

bpe = BPETokenizer()

# train a byte-pair tokenizer on some corpus
train_corpus = "this is some training data. any dumped string will do!"
vocab_size = 8
bpe.train(train_corpus, vocab_size)

# save tokenizer state
bpe.save_encoder("8word.json")

# load tokenizer from dumped file
bpe.load_encoder("8word.json")

# encode and decode
input_ids = bpe.encode("some data")
decoded = bpe.decode(input_ids)

Performance

slightly faster than openai & a lot quicker than 🤗!

alt text

Performance measured on 2.5MB from the wikitext test split using openai's tiktoken gpt2 tokenizer with tiktoken==0.6.0 and the implementation from 🤗 tokenizers at tokenizers==0.19.1

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

toktokenizer-0.2.0.tar.gz (59.2 kB view hashes)

Uploaded Source

Built Distributions

toktokenizer-0.2.0-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ x86-64

toktokenizer-0.2.0-pp310-pypy310_pp73-manylinux_2_17_s390x.manylinux2014_s390x.whl (1.4 MB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ s390x

toktokenizer-0.2.0-pp310-pypy310_pp73-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (1.2 MB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ ppc64le

toktokenizer-0.2.0-pp310-pypy310_pp73-manylinux_2_17_i686.manylinux2014_i686.whl (1.2 MB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ i686

toktokenizer-0.2.0-pp310-pypy310_pp73-manylinux_2_17_armv7l.manylinux2014_armv7l.whl (1.2 MB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ ARMv7l

toktokenizer-0.2.0-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ x86-64

toktokenizer-0.2.0-pp39-pypy39_pp73-manylinux_2_17_s390x.manylinux2014_s390x.whl (1.4 MB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ s390x

toktokenizer-0.2.0-pp39-pypy39_pp73-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (1.2 MB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ ppc64le

toktokenizer-0.2.0-pp39-pypy39_pp73-manylinux_2_17_i686.manylinux2014_i686.whl (1.2 MB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ i686

toktokenizer-0.2.0-pp39-pypy39_pp73-manylinux_2_17_armv7l.manylinux2014_armv7l.whl (1.2 MB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ ARMv7l

toktokenizer-0.2.0-pp39-pypy39_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.2 MB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ ARM64

toktokenizer-0.2.0-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ x86-64

toktokenizer-0.2.0-pp38-pypy38_pp73-manylinux_2_17_s390x.manylinux2014_s390x.whl (1.4 MB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ s390x

toktokenizer-0.2.0-pp38-pypy38_pp73-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (1.2 MB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ ppc64le

toktokenizer-0.2.0-pp38-pypy38_pp73-manylinux_2_17_i686.manylinux2014_i686.whl (1.2 MB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ i686

toktokenizer-0.2.0-pp38-pypy38_pp73-manylinux_2_17_armv7l.manylinux2014_armv7l.whl (1.2 MB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ ARMv7l

toktokenizer-0.2.0-pp38-pypy38_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.2 MB view hashes)

Uploaded PyPy manylinux: glibc 2.17+ ARM64

toktokenizer-0.2.0-cp312-none-win_amd64.whl (237.3 kB view hashes)

Uploaded CPython 3.12 Windows x86-64

toktokenizer-0.2.0-cp312-none-win32.whl (232.0 kB view hashes)

Uploaded CPython 3.12 Windows x86

toktokenizer-0.2.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB view hashes)

Uploaded CPython 3.12 manylinux: glibc 2.17+ x86-64

toktokenizer-0.2.0-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl (1.3 MB view hashes)

Uploaded CPython 3.12 manylinux: glibc 2.17+ s390x

toktokenizer-0.2.0-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (1.2 MB view hashes)

Uploaded CPython 3.12 manylinux: glibc 2.17+ ppc64le

toktokenizer-0.2.0-cp312-cp312-manylinux_2_17_i686.manylinux2014_i686.whl (1.2 MB view hashes)

Uploaded CPython 3.12 manylinux: glibc 2.17+ i686

toktokenizer-0.2.0-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl (1.2 MB view hashes)

Uploaded CPython 3.12 manylinux: glibc 2.17+ ARMv7l

toktokenizer-0.2.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.2 MB view hashes)

Uploaded CPython 3.12 manylinux: glibc 2.17+ ARM64

toktokenizer-0.2.0-cp312-cp312-macosx_11_0_arm64.whl (362.4 kB view hashes)

Uploaded CPython 3.12 macOS 11.0+ ARM64

toktokenizer-0.2.0-cp312-cp312-macosx_10_12_x86_64.whl (370.2 kB view hashes)

Uploaded CPython 3.12 macOS 10.12+ x86-64

toktokenizer-0.2.0-cp311-none-win_amd64.whl (238.0 kB view hashes)

Uploaded CPython 3.11 Windows x86-64

toktokenizer-0.2.0-cp311-none-win32.whl (231.9 kB view hashes)

Uploaded CPython 3.11 Windows x86

toktokenizer-0.2.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64

toktokenizer-0.2.0-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl (1.4 MB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.17+ s390x

toktokenizer-0.2.0-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (1.2 MB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.17+ ppc64le

toktokenizer-0.2.0-cp311-cp311-manylinux_2_17_i686.manylinux2014_i686.whl (1.2 MB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.17+ i686

toktokenizer-0.2.0-cp311-cp311-manylinux_2_17_armv7l.manylinux2014_armv7l.whl (1.2 MB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.17+ ARMv7l

toktokenizer-0.2.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.2 MB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.17+ ARM64

toktokenizer-0.2.0-cp311-cp311-macosx_11_0_arm64.whl (363.7 kB view hashes)

Uploaded CPython 3.11 macOS 11.0+ ARM64

toktokenizer-0.2.0-cp311-cp311-macosx_10_12_x86_64.whl (372.4 kB view hashes)

Uploaded CPython 3.11 macOS 10.12+ x86-64

toktokenizer-0.2.0-cp310-none-win_amd64.whl (238.3 kB view hashes)

Uploaded CPython 3.10 Windows x86-64

toktokenizer-0.2.0-cp310-none-win32.whl (232.1 kB view hashes)

Uploaded CPython 3.10 Windows x86

toktokenizer-0.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

toktokenizer-0.2.0-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl (1.4 MB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.17+ s390x

toktokenizer-0.2.0-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (1.2 MB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.17+ ppc64le

toktokenizer-0.2.0-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl (1.2 MB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.17+ i686

toktokenizer-0.2.0-cp310-cp310-manylinux_2_17_armv7l.manylinux2014_armv7l.whl (1.2 MB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.17+ ARMv7l

toktokenizer-0.2.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.2 MB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.17+ ARM64

toktokenizer-0.2.0-cp310-cp310-macosx_11_0_arm64.whl (363.6 kB view hashes)

Uploaded CPython 3.10 macOS 11.0+ ARM64

toktokenizer-0.2.0-cp310-cp310-macosx_10_12_x86_64.whl (372.2 kB view hashes)

Uploaded CPython 3.10 macOS 10.12+ x86-64

toktokenizer-0.2.0-cp39-none-win_amd64.whl (238.0 kB view hashes)

Uploaded CPython 3.9 Windows x86-64

toktokenizer-0.2.0-cp39-none-win32.whl (232.4 kB view hashes)

Uploaded CPython 3.9 Windows x86

toktokenizer-0.2.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

toktokenizer-0.2.0-cp39-cp39-manylinux_2_17_s390x.manylinux2014_s390x.whl (1.4 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.17+ s390x

toktokenizer-0.2.0-cp39-cp39-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (1.2 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ppc64le

toktokenizer-0.2.0-cp39-cp39-manylinux_2_17_i686.manylinux2014_i686.whl (1.2 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.17+ i686

toktokenizer-0.2.0-cp39-cp39-manylinux_2_17_armv7l.manylinux2014_armv7l.whl (1.2 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ARMv7l

toktokenizer-0.2.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.2 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ARM64

toktokenizer-0.2.0-cp39-cp39-macosx_11_0_arm64.whl (364.0 kB view hashes)

Uploaded CPython 3.9 macOS 11.0+ ARM64

toktokenizer-0.2.0-cp39-cp39-macosx_10_12_x86_64.whl (372.6 kB view hashes)

Uploaded CPython 3.9 macOS 10.12+ x86-64

toktokenizer-0.2.0-cp38-none-win_amd64.whl (237.9 kB view hashes)

Uploaded CPython 3.8 Windows x86-64

toktokenizer-0.2.0-cp38-none-win32.whl (232.9 kB view hashes)

Uploaded CPython 3.8 Windows x86

toktokenizer-0.2.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

toktokenizer-0.2.0-cp38-cp38-manylinux_2_17_s390x.manylinux2014_s390x.whl (1.4 MB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.17+ s390x

toktokenizer-0.2.0-cp38-cp38-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (1.2 MB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.17+ ppc64le

toktokenizer-0.2.0-cp38-cp38-manylinux_2_17_i686.manylinux2014_i686.whl (1.2 MB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.17+ i686

toktokenizer-0.2.0-cp38-cp38-manylinux_2_17_armv7l.manylinux2014_armv7l.whl (1.2 MB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.17+ ARMv7l

toktokenizer-0.2.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.2 MB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.17+ ARM64

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page