No project description provided
Project description
Gigatoken
400x faster than HuggingFace, drop-in replacement.
Tokenize your text data at GB/s!
Install
pip install gigatoken
Usage
import gigatoken as gt
hf_tokenizer = ...
How does Gigatoken work?
Gigatoken came from a few observations:
- A majority of the time in current tokenizers is spent on pretokenization ->
Gigatoken implements pretokenizers that run at >2GB/s/thread
Gigatoken is faster than other libraries due to algorithmic and systems changes. One key benefit of using Gigatoken is that it replaces the regex expression used by almost all tokenizers to do pre-tokenization with a custom implementation of the exact same method. This is a serious bottleneck for other implementations, and is a big part in this library's breakneck speeds. Additionally, Gigatoken uses concurrent data structures to use multiprocessing in more places.
* All reference speeds in this section are measured on an M4 Pro CPU
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file gigatoken-0.3.0.tar.gz.
File metadata
- Download URL: gigatoken-0.3.0.tar.gz
- Upload date:
- Size: 519.1 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: uv/0.11.28 {"installer":{"name":"uv","version":"0.11.28","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
e40e83350016d7b8141f34177c3f68db4e8db24e3d8b54860985dba01587192a
|
|
| MD5 |
1243d7c73d732e2b2ea45c6c457f96fc
|
|
| BLAKE2b-256 |
e64d55a33482d445e9e5545bc9219fec774b47089c1f6cac764fcde870f61b69
|
File details
Details for the file gigatoken-0.3.0-cp310-abi3-win_arm64.whl.
File metadata
- Download URL: gigatoken-0.3.0-cp310-abi3-win_arm64.whl
- Upload date:
- Size: 1.1 MB
- Tags: CPython 3.10+, Windows ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via: uv/0.11.28 {"installer":{"name":"uv","version":"0.11.28","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
9c901223f38b5b184cd625ccb5c2cec9d1042a88d3a9f8cb88bfe7d9220578bf
|
|
| MD5 |
fe361d57b6477e76f9852d2da7d5f2a7
|
|
| BLAKE2b-256 |
531ff26ab5b25f7667c9572dab502a534da4335fcd7e275c328614a74bdf46da
|
File details
Details for the file gigatoken-0.3.0-cp310-abi3-win_amd64.whl.
File metadata
- Download URL: gigatoken-0.3.0-cp310-abi3-win_amd64.whl
- Upload date:
- Size: 1.2 MB
- Tags: CPython 3.10+, Windows x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: uv/0.11.28 {"installer":{"name":"uv","version":"0.11.28","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
f61e2f8e72b2239d5aa82b4d194be12c484dd69a374f45d2d0e6669314fb88d1
|
|
| MD5 |
9513f668f33a724d94b79ea9810ffe47
|
|
| BLAKE2b-256 |
0724bf01343024bdcf3a40a10976f6eeec7c87526311c7e609ebab02ac66828b
|
File details
Details for the file gigatoken-0.3.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.
File metadata
- Download URL: gigatoken-0.3.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- Upload date:
- Size: 1.4 MB
- Tags: CPython 3.10+, manylinux: glibc 2.17+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: uv/0.11.28 {"installer":{"name":"uv","version":"0.11.28","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
8f0ed9042e00e19c7e4fc96cdf2722e7ea7360e44f71bc4a8813efaa5d65c7f8
|
|
| MD5 |
060914612a76ecb7b8c40d3edb805e1f
|
|
| BLAKE2b-256 |
9d8b6b39c84fb7f56fa61696e8eef34177072609f28073ad18b93bd4d7cf8b04
|
File details
Details for the file gigatoken-0.3.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.
File metadata
- Download URL: gigatoken-0.3.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
- Upload date:
- Size: 1.3 MB
- Tags: CPython 3.10+, manylinux: glibc 2.17+ ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via: uv/0.11.28 {"installer":{"name":"uv","version":"0.11.28","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
2123b9a43ed8fb02333e5ccfbe7adb2e1c273c9aeade0c8a664e73fa0f3b8f2e
|
|
| MD5 |
6f0e53fb9d38663defa4f4cab80678a9
|
|
| BLAKE2b-256 |
6b1a5c47ab57f8a9453f689701de651fa1f2689206b1efb0201c303a8686c3d2
|
File details
Details for the file gigatoken-0.3.0-cp310-abi3-macosx_11_0_arm64.whl.
File metadata
- Download URL: gigatoken-0.3.0-cp310-abi3-macosx_11_0_arm64.whl
- Upload date:
- Size: 1.2 MB
- Tags: CPython 3.10+, macOS 11.0+ ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via: uv/0.11.28 {"installer":{"name":"uv","version":"0.11.28","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
e7768f73b6d5a195a13c1aab85cdbaa40ca3312b60a89ca740e53b96003e7e60
|
|
| MD5 |
4c2b1aa944fc975f4205ae5af77ed81a
|
|
| BLAKE2b-256 |
0bd8ee053f6fc08e4d79971f4ce1d76330ae0ebe5e73050ede8385f23083d4b7
|
File details
Details for the file gigatoken-0.3.0-cp310-abi3-macosx_10_12_x86_64.whl.
File metadata
- Download URL: gigatoken-0.3.0-cp310-abi3-macosx_10_12_x86_64.whl
- Upload date:
- Size: 1.3 MB
- Tags: CPython 3.10+, macOS 10.12+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: uv/0.11.28 {"installer":{"name":"uv","version":"0.11.28","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
b1c6535d99a636ec3a204cf66bc3ef186f5923584a37cdc976c92849acdc9e7f
|
|
| MD5 |
bfed24bb4c67db9b4d64a07ecd81cbf0
|
|
| BLAKE2b-256 |
02819bdba321aad6544c67a2fa1c3e30c988975b5bd40690dafe782d056c91fa
|
