gpt3_tokenizer
An OpenAI GPT3 helper library for encoding/decoding strings and counting tokens.
Counting tokens gives the same output as OpenAI’s tokenizer
Supported python versions: >=2.7 <3.0 OR >=3.3
Installing
pip install gpt3_tokenizerExamples
Encoding/decoding a string
import gpt3_tokenizer
a_string = "That's my beautiful and sweet string"
encoded = gpt3_tokenizer.encode(a_string) # outputs [2504, 338, 616, 4950, 290, 6029, 4731]
decoded = gpt3_tokenizer.decode(encoded) # outputs "That's my beautiful and sweet string"Counting tokens
import gpt3_tokenizer
a_string = "That's my beautiful and sweet string"
tokens_count = gpt3_tokenizer.count_tokens(a_string) # outputs 71 maintainer
Metadata
1 maintainer
Release files for gpt3_tokenizer 0.1.5
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| gpt3_tokenizer-0.1.5.tar.gz | 560.7 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| gpt3_tokenizer-0.1.5-py2.py3-none-any.whl | Python 3, Python 2 | none | any | Details |
Total release size: 1.1 MB
Release files / gpt3_tokenizer-0.1.5.tar.gz
| Download URL | gpt3_tokenizer-0.1.5.tar.gz |
|---|---|
| Size | 560.7 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
0366a9b7000b1a9066ae9257eed7f2b665b374788d38bbeac86ab9145ab1a6c9
|
|
BLAKE2b-256 checksum How to use checksums |
46f633582154322e1444ba6a677a4bca1d79e1221304e3607b14881c3510bf56
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.8.2 CPython/3.11.3 Darwin/23.3.0
|
Release files / gpt3_tokenizer-0.1.5-py2.py3-none-any.whl
| Download URL | gpt3_tokenizer-0.1.5-py2.py3-none-any.whl |
|---|---|
| Size | 567.8 kB |
| Tags | Python 2 Python 3 |
|
SHA-256 checksum How to use checksums |
2d0ed9c7efa907d45ce3c338ffe2ee3bc9124ee1236248989bd883fd4eb0e5b6
|
|
BLAKE2b-256 checksum How to use checksums |
ebedaf04d6badad07846809ef832ab1ea294697961d82c2c330445bd081b1e47
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
poetry/1.8.2 CPython/3.11.3 Darwin/23.3.0
|
