Skip to main content

No project description provided

Project description



Build GitHub


Tokenizers

Provides an implementation of today's most used tokenizers, with a focus on performance and versatility.

Bindings over the Rust implementation. If you are interested in the High-level design, you can go check it there.

Otherwise, let's dive in!

Main features:

  • Train new vocabularies and tokenize using 4 pre-made tokenizers (Bert WordPiece and the 3 most common BPE versions).
  • Extremely fast (both training and tokenization), thanks to the Rust implementation. Takes less than 20 seconds to tokenize a GB of text on a server's CPU.
  • Easy to use, but also extremely versatile.
  • Designed for research and production.
  • Normalization comes with alignments tracking. It's always possible to get the part of the original sentence that corresponds to a given token.
  • Does all the pre-processing: Truncate, Pad, add the special tokens your model needs.

Installation

With pip:

pip install tokenizers

From sources:

To use this method, you need to have the Rust installed:

# Install with:
curl https://sh.rustup.rs -sSf | sh -s -- -y
export PATH="$HOME/.cargo/bin:$PATH"

Once Rust is installed, you can compile doing the following

git clone https://github.com/huggingface/tokenizers
cd tokenizers/bindings/python

# Create a virtual env (you can use yours as well)
python -m venv .env
source .env/bin/activate

# Install `tokenizers` in the current virtual env
pip install -e .

Load a pretrained tokenizer from the Hub

from tokenizers import Tokenizer

tokenizer = Tokenizer.from_pretrained("bert-base-cased")

Using the provided Tokenizers

We provide some pre-build tokenizers to cover the most common cases. You can easily load one of these using some vocab.json and merges.txt files:

from tokenizers import CharBPETokenizer

# Initialize a tokenizer
vocab = "./path/to/vocab.json"
merges = "./path/to/merges.txt"
tokenizer = CharBPETokenizer(vocab, merges)

# And then encode:
encoded = tokenizer.encode("I can feel the magic, can you?")
print(encoded.ids)
print(encoded.tokens)

And you can train them just as simply:

from tokenizers import CharBPETokenizer

# Initialize a tokenizer
tokenizer = CharBPETokenizer()

# Then train it!
tokenizer.train([ "./path/to/files/1.txt", "./path/to/files/2.txt" ])

# Now, let's use it:
encoded = tokenizer.encode("I can feel the magic, can you?")

# And finally save it somewhere
tokenizer.save("./path/to/directory/my-bpe.tokenizer.json")

Provided Tokenizers

  • CharBPETokenizer: The original BPE
  • ByteLevelBPETokenizer: The byte level version of the BPE
  • SentencePieceBPETokenizer: A BPE implementation compatible with the one used by SentencePiece
  • BertWordPieceTokenizer: The famous Bert tokenizer, using WordPiece

All of these can be used and trained as explained above!

Build your own

Whenever these provided tokenizers don't give you enough freedom, you can build your own tokenizer, by putting all the different parts you need together. You can check how we implemented the provided tokenizers and adapt them easily to your own needs.

Building a byte-level BPE

Here is an example showing how to build your own byte-level BPE by putting all the different pieces together, and then saving it to a single file:

from tokenizers import Tokenizer, models, pre_tokenizers, decoders, trainers, processors

# Initialize a tokenizer
tokenizer = Tokenizer(models.BPE())

# Customize pre-tokenization and decoding
tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=True)
tokenizer.decoder = decoders.ByteLevel()
tokenizer.post_processor = processors.ByteLevel(trim_offsets=True)

# And then train
trainer = trainers.BpeTrainer(
    vocab_size=20000,
    min_frequency=2,
    initial_alphabet=pre_tokenizers.ByteLevel.alphabet()
)
tokenizer.train([
    "./path/to/dataset/1.txt",
    "./path/to/dataset/2.txt",
    "./path/to/dataset/3.txt"
], trainer=trainer)

# And Save it
tokenizer.save("byte-level-bpe.tokenizer.json", pretty=True)

Now, when you want to use this tokenizer, this is as simple as:

from tokenizers import Tokenizer

tokenizer = Tokenizer.from_file("byte-level-bpe.tokenizer.json")

encoded = tokenizer.encode("I can feel the magic, can you?")

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

tokenizers_gt-0.15.0.post1.tar.gz (322.3 kB view hashes)

Uploaded Source

Built Distributions

tokenizers_gt-0.15.0.post1-pp310-pypy310_pp73-musllinux_1_1_x86_64.whl (10.2 MB view hashes)

Uploaded PyPy musllinux: musl 1.1+ x86-64

tokenizers_gt-0.15.0.post1-pp310-pypy310_pp73-musllinux_1_1_aarch64.whl (9.9 MB view hashes)

Uploaded PyPy musllinux: musl 1.1+ ARM64

tokenizers_gt-0.15.0.post1-pp310-pypy310_pp73-macosx_11_0_arm64.whl (2.5 MB view hashes)

Uploaded PyPy macOS 11.0+ ARM64

tokenizers_gt-0.15.0.post1-pp310-pypy310_pp73-macosx_10_7_x86_64.whl (2.6 MB view hashes)

Uploaded PyPy macOS 10.7+ x86-64

tokenizers_gt-0.15.0.post1-pp39-pypy39_pp73-musllinux_1_1_x86_64.whl (10.2 MB view hashes)

Uploaded PyPy musllinux: musl 1.1+ x86-64

tokenizers_gt-0.15.0.post1-pp39-pypy39_pp73-musllinux_1_1_aarch64.whl (9.9 MB view hashes)

Uploaded PyPy musllinux: musl 1.1+ ARM64

tokenizers_gt-0.15.0.post1-pp39-pypy39_pp73-macosx_11_0_arm64.whl (2.5 MB view hashes)

Uploaded PyPy macOS 11.0+ ARM64

tokenizers_gt-0.15.0.post1-pp39-pypy39_pp73-macosx_10_7_x86_64.whl (2.6 MB view hashes)

Uploaded PyPy macOS 10.7+ x86-64

tokenizers_gt-0.15.0.post1-pp38-pypy38_pp73-musllinux_1_1_x86_64.whl (10.2 MB view hashes)

Uploaded PyPy musllinux: musl 1.1+ x86-64

tokenizers_gt-0.15.0.post1-pp38-pypy38_pp73-musllinux_1_1_aarch64.whl (9.9 MB view hashes)

Uploaded PyPy musllinux: musl 1.1+ ARM64

tokenizers_gt-0.15.0.post1-pp38-pypy38_pp73-macosx_11_0_arm64.whl (2.5 MB view hashes)

Uploaded PyPy macOS 11.0+ ARM64

tokenizers_gt-0.15.0.post1-pp38-pypy38_pp73-macosx_10_7_x86_64.whl (2.6 MB view hashes)

Uploaded PyPy macOS 10.7+ x86-64

tokenizers_gt-0.15.0.post1-pp37-pypy37_pp73-musllinux_1_1_x86_64.whl (10.2 MB view hashes)

Uploaded PyPy musllinux: musl 1.1+ x86-64

tokenizers_gt-0.15.0.post1-pp37-pypy37_pp73-musllinux_1_1_aarch64.whl (9.9 MB view hashes)

Uploaded PyPy musllinux: musl 1.1+ ARM64

tokenizers_gt-0.15.0.post1-pp37-pypy37_pp73-macosx_10_7_x86_64.whl (2.6 MB view hashes)

Uploaded PyPy macOS 10.7+ x86-64

tokenizers_gt-0.15.0.post1-cp312-cp312-musllinux_1_1_x86_64.whl (10.2 MB view hashes)

Uploaded CPython 3.12 musllinux: musl 1.1+ x86-64

tokenizers_gt-0.15.0.post1-cp312-cp312-musllinux_1_1_aarch64.whl (9.9 MB view hashes)

Uploaded CPython 3.12 musllinux: musl 1.1+ ARM64

tokenizers_gt-0.15.0.post1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.9 MB view hashes)

Uploaded CPython 3.12 manylinux: glibc 2.17+ x86-64

tokenizers_gt-0.15.0.post1-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl (4.3 MB view hashes)

Uploaded CPython 3.12 manylinux: glibc 2.17+ s390x

tokenizers_gt-0.15.0.post1-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (4.3 MB view hashes)

Uploaded CPython 3.12 manylinux: glibc 2.17+ ppc64le

tokenizers_gt-0.15.0.post1-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl (3.7 MB view hashes)

Uploaded CPython 3.12 manylinux: glibc 2.17+ ARMv7l

tokenizers_gt-0.15.0.post1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (3.8 MB view hashes)

Uploaded CPython 3.12 manylinux: glibc 2.17+ ARM64

tokenizers_gt-0.15.0.post1-cp312-cp312-manylinux_2_12_i686.manylinux2010_i686.whl (3.9 MB view hashes)

Uploaded CPython 3.12 manylinux: glibc 2.12+ i686

tokenizers_gt-0.15.0.post1-cp312-cp312-macosx_11_0_arm64.whl (2.5 MB view hashes)

Uploaded CPython 3.12 macOS 11.0+ ARM64

tokenizers_gt-0.15.0.post1-cp312-cp312-macosx_10_7_x86_64.whl (2.6 MB view hashes)

Uploaded CPython 3.12 macOS 10.7+ x86-64

tokenizers_gt-0.15.0.post1-cp311-none-win_amd64.whl (2.2 MB view hashes)

Uploaded CPython 3.11 Windows x86-64

tokenizers_gt-0.15.0.post1-cp311-none-win32.whl (2.0 MB view hashes)

Uploaded CPython 3.11 Windows x86

tokenizers_gt-0.15.0.post1-cp311-cp311-musllinux_1_1_x86_64.whl (10.2 MB view hashes)

Uploaded CPython 3.11 musllinux: musl 1.1+ x86-64

tokenizers_gt-0.15.0.post1-cp311-cp311-musllinux_1_1_aarch64.whl (9.9 MB view hashes)

Uploaded CPython 3.11 musllinux: musl 1.1+ ARM64

tokenizers_gt-0.15.0.post1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.9 MB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.17+ x86-64

tokenizers_gt-0.15.0.post1-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl (4.4 MB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.17+ s390x

tokenizers_gt-0.15.0.post1-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (4.3 MB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.17+ ppc64le

tokenizers_gt-0.15.0.post1-cp311-cp311-manylinux_2_17_armv7l.manylinux2014_armv7l.whl (3.7 MB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.17+ ARMv7l

tokenizers_gt-0.15.0.post1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (3.8 MB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.17+ ARM64

tokenizers_gt-0.15.0.post1-cp311-cp311-manylinux_2_12_i686.manylinux2010_i686.whl (3.9 MB view hashes)

Uploaded CPython 3.11 manylinux: glibc 2.12+ i686

tokenizers_gt-0.15.0.post1-cp311-cp311-macosx_11_0_arm64.whl (2.5 MB view hashes)

Uploaded CPython 3.11 macOS 11.0+ ARM64

tokenizers_gt-0.15.0.post1-cp311-cp311-macosx_10_7_x86_64.whl (2.6 MB view hashes)

Uploaded CPython 3.11 macOS 10.7+ x86-64

tokenizers_gt-0.15.0.post1-cp310-none-win_amd64.whl (2.2 MB view hashes)

Uploaded CPython 3.10 Windows x86-64

tokenizers_gt-0.15.0.post1-cp310-none-win32.whl (2.0 MB view hashes)

Uploaded CPython 3.10 Windows x86

tokenizers_gt-0.15.0.post1-cp310-cp310-musllinux_1_1_x86_64.whl (10.2 MB view hashes)

Uploaded CPython 3.10 musllinux: musl 1.1+ x86-64

tokenizers_gt-0.15.0.post1-cp310-cp310-musllinux_1_1_aarch64.whl (9.9 MB view hashes)

Uploaded CPython 3.10 musllinux: musl 1.1+ ARM64

tokenizers_gt-0.15.0.post1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.9 MB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.17+ x86-64

tokenizers_gt-0.15.0.post1-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl (4.4 MB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.17+ s390x

tokenizers_gt-0.15.0.post1-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (4.3 MB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.17+ ppc64le

tokenizers_gt-0.15.0.post1-cp310-cp310-manylinux_2_17_armv7l.manylinux2014_armv7l.whl (3.7 MB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.17+ ARMv7l

tokenizers_gt-0.15.0.post1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (3.8 MB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.17+ ARM64

tokenizers_gt-0.15.0.post1-cp310-cp310-manylinux_2_12_i686.manylinux2010_i686.whl (3.9 MB view hashes)

Uploaded CPython 3.10 manylinux: glibc 2.12+ i686

tokenizers_gt-0.15.0.post1-cp310-cp310-macosx_11_0_arm64.whl (2.5 MB view hashes)

Uploaded CPython 3.10 macOS 11.0+ ARM64

tokenizers_gt-0.15.0.post1-cp310-cp310-macosx_10_7_x86_64.whl (2.6 MB view hashes)

Uploaded CPython 3.10 macOS 10.7+ x86-64

tokenizers_gt-0.15.0.post1-cp39-none-win_amd64.whl (2.2 MB view hashes)

Uploaded CPython 3.9 Windows x86-64

tokenizers_gt-0.15.0.post1-cp39-none-win32.whl (2.0 MB view hashes)

Uploaded CPython 3.9 Windows x86

tokenizers_gt-0.15.0.post1-cp39-cp39-musllinux_1_1_x86_64.whl (10.2 MB view hashes)

Uploaded CPython 3.9 musllinux: musl 1.1+ x86-64

tokenizers_gt-0.15.0.post1-cp39-cp39-musllinux_1_1_aarch64.whl (9.9 MB view hashes)

Uploaded CPython 3.9 musllinux: musl 1.1+ ARM64

tokenizers_gt-0.15.0.post1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.9 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.17+ x86-64

tokenizers_gt-0.15.0.post1-cp39-cp39-manylinux_2_17_s390x.manylinux2014_s390x.whl (4.4 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.17+ s390x

tokenizers_gt-0.15.0.post1-cp39-cp39-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (4.3 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ppc64le

tokenizers_gt-0.15.0.post1-cp39-cp39-manylinux_2_17_armv7l.manylinux2014_armv7l.whl (3.7 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ARMv7l

tokenizers_gt-0.15.0.post1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (3.8 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ARM64

tokenizers_gt-0.15.0.post1-cp39-cp39-manylinux_2_12_i686.manylinux2010_i686.whl (3.9 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.12+ i686

tokenizers_gt-0.15.0.post1-cp39-cp39-macosx_11_0_arm64.whl (2.5 MB view hashes)

Uploaded CPython 3.9 macOS 11.0+ ARM64

tokenizers_gt-0.15.0.post1-cp39-cp39-macosx_10_7_x86_64.whl (2.6 MB view hashes)

Uploaded CPython 3.9 macOS 10.7+ x86-64

tokenizers_gt-0.15.0.post1-cp38-none-win_amd64.whl (2.2 MB view hashes)

Uploaded CPython 3.8 Windows x86-64

tokenizers_gt-0.15.0.post1-cp38-none-win32.whl (2.0 MB view hashes)

Uploaded CPython 3.8 Windows x86

tokenizers_gt-0.15.0.post1-cp38-cp38-musllinux_1_1_x86_64.whl (10.2 MB view hashes)

Uploaded CPython 3.8 musllinux: musl 1.1+ x86-64

tokenizers_gt-0.15.0.post1-cp38-cp38-musllinux_1_1_aarch64.whl (9.9 MB view hashes)

Uploaded CPython 3.8 musllinux: musl 1.1+ ARM64

tokenizers_gt-0.15.0.post1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.9 MB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.17+ x86-64

tokenizers_gt-0.15.0.post1-cp38-cp38-manylinux_2_17_s390x.manylinux2014_s390x.whl (4.4 MB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.17+ s390x

tokenizers_gt-0.15.0.post1-cp38-cp38-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (4.3 MB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.17+ ppc64le

tokenizers_gt-0.15.0.post1-cp38-cp38-manylinux_2_17_armv7l.manylinux2014_armv7l.whl (3.7 MB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.17+ ARMv7l

tokenizers_gt-0.15.0.post1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (3.8 MB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.17+ ARM64

tokenizers_gt-0.15.0.post1-cp38-cp38-manylinux_2_12_i686.manylinux2010_i686.whl (3.9 MB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.12+ i686

tokenizers_gt-0.15.0.post1-cp38-cp38-macosx_11_0_arm64.whl (2.5 MB view hashes)

Uploaded CPython 3.8 macOS 11.0+ ARM64

tokenizers_gt-0.15.0.post1-cp38-cp38-macosx_10_7_x86_64.whl (2.6 MB view hashes)

Uploaded CPython 3.8 macOS 10.7+ x86-64

tokenizers_gt-0.15.0.post1-cp37-none-win_amd64.whl (2.2 MB view hashes)

Uploaded CPython 3.7 Windows x86-64

tokenizers_gt-0.15.0.post1-cp37-none-win32.whl (2.0 MB view hashes)

Uploaded CPython 3.7 Windows x86

tokenizers_gt-0.15.0.post1-cp37-cp37m-musllinux_1_1_x86_64.whl (10.2 MB view hashes)

Uploaded CPython 3.7m musllinux: musl 1.1+ x86-64

tokenizers_gt-0.15.0.post1-cp37-cp37m-musllinux_1_1_aarch64.whl (9.9 MB view hashes)

Uploaded CPython 3.7m musllinux: musl 1.1+ ARM64

tokenizers_gt-0.15.0.post1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.9 MB view hashes)

Uploaded CPython 3.7m manylinux: glibc 2.17+ x86-64

tokenizers_gt-0.15.0.post1-cp37-cp37m-manylinux_2_17_s390x.manylinux2014_s390x.whl (4.4 MB view hashes)

Uploaded CPython 3.7m manylinux: glibc 2.17+ s390x

tokenizers_gt-0.15.0.post1-cp37-cp37m-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (4.3 MB view hashes)

Uploaded CPython 3.7m manylinux: glibc 2.17+ ppc64le

tokenizers_gt-0.15.0.post1-cp37-cp37m-manylinux_2_17_armv7l.manylinux2014_armv7l.whl (3.7 MB view hashes)

Uploaded CPython 3.7m manylinux: glibc 2.17+ ARMv7l

tokenizers_gt-0.15.0.post1-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (3.8 MB view hashes)

Uploaded CPython 3.7m manylinux: glibc 2.17+ ARM64

tokenizers_gt-0.15.0.post1-cp37-cp37m-manylinux_2_12_i686.manylinux2010_i686.whl (3.9 MB view hashes)

Uploaded CPython 3.7m manylinux: glibc 2.12+ i686

tokenizers_gt-0.15.0.post1-cp37-cp37m-macosx_11_0_arm64.whl (2.5 MB view hashes)

Uploaded CPython 3.7m macOS 11.0+ ARM64

tokenizers_gt-0.15.0.post1-cp37-cp37m-macosx_10_7_x86_64.whl (2.6 MB view hashes)

Uploaded CPython 3.7m macOS 10.7+ x86-64

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page