Tokenize2
Tokenize2 is an improved byte-level BPE tokenizer, inspired by models like GPT-3, designed for efficient tokenization of text into subword units. It supports special tokens and byte-level text handling for robust tokenization, including for non-ASCII characters.
Features
- Byte-level tokenization for handling a wide range of characters
- Special tokens (like
<PAD>,<UNK>) for flexible token management - Supports efficient BPE merges for subword tokenization
- Suitable for natural language processing and text generation tasks
Installation
You can install Tokenize2 via pip:
1 maintainer
Metadata
1 maintainer
Release files for Tokenize2 2.0.3
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| tokenize2-2.0.3.tar.gz | 3.9 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| Tokenize2-2.0.3-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 8.0 kB
Release files / tokenize2-2.0.3.tar.gz
| Download URL | tokenize2-2.0.3.tar.gz |
|---|---|
| Size | 3.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
61c5730bc4d897cb55dae1c9a00b05ffc66d7c794623f75f33d6e69f7c7a6d86
|
|
BLAKE2b-256 checksum How to use checksums |
231e19f970ac89f7b6dcbafa5aed933331fd2a57cb989c9ecc7f756f15d3df40
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.1.1 CPython/3.12.3
|
Release files / Tokenize2-2.0.3-py3-none-any.whl
| Download URL | Tokenize2-2.0.3-py3-none-any.whl |
|---|---|
| Size | 4.2 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
6f58195d600e503d295bdbf650ce2b58cee4de0e7bd3ebc773ffa0350e492ecc
|
|
BLAKE2b-256 checksum How to use checksums |
b4a0dba0a280c36708e5c1870dc1d8cb6b8eccb31287a9fadbcbc4ccc73a1276
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.1.1 CPython/3.12.3
|
