No project description provided
Project description
Gigatok
400x faster than HuggingFace, drop-in replacement.
Tokenize your text data at GB/s!
Install
pip install gigatok
How does Gigatok work?
Gigatok came from a few observations:
- A majority of the time in current tokenizers is spent on pretokenization ->
Gigatok implements pretokenizers that run at >2GB/s/thread
Gigatok is faster than other libraries due to algorithmic and systems changes. One key benefit of using Gigatok is that it replaces the regex expression used by almost all tokenizers to do pre-tokenization with a custom implementation of the exact same method. This is a serious bottleneck for other implementations, and is a big part in this library's breakneck speeds. Additionally, Gigatok uses concurrent data structures to use multiprocessing in more places.
* All reference speeds in this section are measured on an M4 Pro CPU
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file gigatok-0.2.0.tar.gz.
File metadata
- Download URL: gigatok-0.2.0.tar.gz
- Upload date:
- Size: 513.4 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: uv/0.11.23 {"installer":{"name":"uv","version":"0.11.23","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
3c0037211a4b91e6105936f5a0829945108bb9429dd911c46b464687aa573695
|
|
| MD5 |
0ec8ac946d930445139b01a845647a9e
|
|
| BLAKE2b-256 |
e78e83a3c5156edb72df1f377a93dbfbb955d9ebd5ef2ef0e2293d2eea07ae96
|
File details
Details for the file gigatok-0.2.0-cp310-abi3-win_arm64.whl.
File metadata
- Download URL: gigatok-0.2.0-cp310-abi3-win_arm64.whl
- Upload date:
- Size: 1.1 MB
- Tags: CPython 3.10+, Windows ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via: uv/0.11.23 {"installer":{"name":"uv","version":"0.11.23","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
9b19277cd981d5197f24075371558ff503a3747b8362c9429963d7cfdaea56e3
|
|
| MD5 |
96437bf7730e90fa8796b89d156382d3
|
|
| BLAKE2b-256 |
ae96bdc82d0fa40a67f0284025b6f2ee96cf30fba6d7c92e6aa0156097e9253b
|
File details
Details for the file gigatok-0.2.0-cp310-abi3-win_amd64.whl.
File metadata
- Download URL: gigatok-0.2.0-cp310-abi3-win_amd64.whl
- Upload date:
- Size: 1.2 MB
- Tags: CPython 3.10+, Windows x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: uv/0.11.23 {"installer":{"name":"uv","version":"0.11.23","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
892631f8e72e1a9f0596161cfa528bffe13840db4de973ec5b44ec5557ea431f
|
|
| MD5 |
edbf8886d5304ab2f535ed92f1299426
|
|
| BLAKE2b-256 |
a5200b8438ed52d6b31b03af84fa4e66bc3151d019941e884194bbacecdc1c45
|
File details
Details for the file gigatok-0.2.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.
File metadata
- Download URL: gigatok-0.2.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- Upload date:
- Size: 1.4 MB
- Tags: CPython 3.10+, manylinux: glibc 2.17+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: uv/0.11.23 {"installer":{"name":"uv","version":"0.11.23","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
feb785c208851502cf19c7a78f85b63c50f1a85f2e05cff1bd19c2486e4c7b98
|
|
| MD5 |
4fb9526f720a5e3500dc5a70cd229dcf
|
|
| BLAKE2b-256 |
ec61cd7611e8a883279cd094d26e47d9741be852ef5fa2a95820fe71d444e2e5
|
File details
Details for the file gigatok-0.2.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.
File metadata
- Download URL: gigatok-0.2.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
- Upload date:
- Size: 1.2 MB
- Tags: CPython 3.10+, manylinux: glibc 2.17+ ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via: uv/0.11.23 {"installer":{"name":"uv","version":"0.11.23","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
e38347277d9051a177c58ffdbbb2755a51003ccfc36fd7cbbf84924768794fbc
|
|
| MD5 |
c157d64c6913177ae119b6efcf69bee6
|
|
| BLAKE2b-256 |
892aecae97bda39e0e55ce1d945c7c951476caddb195a8b6e708ff32f00b539e
|
File details
Details for the file gigatok-0.2.0-cp310-abi3-macosx_11_0_arm64.whl.
File metadata
- Download URL: gigatok-0.2.0-cp310-abi3-macosx_11_0_arm64.whl
- Upload date:
- Size: 1.2 MB
- Tags: CPython 3.10+, macOS 11.0+ ARM64
- Uploaded using Trusted Publishing? No
- Uploaded via: uv/0.11.23 {"installer":{"name":"uv","version":"0.11.23","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
9822eb118cfb256a02b54910327a8fdb9fa7ff023843e4ab5ed231922a5d05b0
|
|
| MD5 |
b708c5d909dda246d3e4c894e994b14e
|
|
| BLAKE2b-256 |
de871856eb081ab4c09406a0d2559dfc3bdec56dc6a010e4036b3ca5611ff2c9
|
File details
Details for the file gigatok-0.2.0-cp310-abi3-macosx_10_12_x86_64.whl.
File metadata
- Download URL: gigatok-0.2.0-cp310-abi3-macosx_10_12_x86_64.whl
- Upload date:
- Size: 1.3 MB
- Tags: CPython 3.10+, macOS 10.12+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: uv/0.11.23 {"installer":{"name":"uv","version":"0.11.23","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"macOS","version":null,"id":null,"libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":null}
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
a89e3eb5c8cfb1f42b79b84e7927d80ca918b7980496de6b00f11728a483302b
|
|
| MD5 |
7a93dd0f116b16f4d656eafc1f8396d1
|
|
| BLAKE2b-256 |
e6646c68c72f401131e94713fe58203e18bb5aafed02fd91c3a69278183cdb98
|
