A command-line utility for working with BPE tokenizers
Project description
tkn
tkn is a command-line utility to quickly tokenize with tiktoken.
Installation
pip install tkn
Example usage:
$ ls
document_1.txt
document_2.txt
$ tkn document_1.txt
[tokenized version of the data]
$ tkn document_1.txt -s '\n' | wc -l
2094 # document contains 2094 tokens
$ tkn -m gpt-4 document_1.txt | tkn -m gpt-4 -d
[the contents of document_1.txt]
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
tkn-0.1.2.tar.gz
(1.9 kB
view details)
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
tkn-0.1.2-py3-none-any.whl
(2.5 kB
view details)
File details
Details for the file tkn-0.1.2.tar.gz.
File metadata
- Download URL: tkn-0.1.2.tar.gz
- Upload date:
- Size: 1.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.6.1 CPython/3.11.5 Darwin/22.6.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
01d08f3b62c73b04b7a109d287a11772aa4ed38bbac469d7ebaf4d676ace82af
|
|
| MD5 |
69b9ea7b03b368259fe335362b362e5f
|
|
| BLAKE2b-256 |
84a21baf353f609fd4cf06f03431baa9448997710a88287aec0b7b35153d0014
|
File details
Details for the file tkn-0.1.2-py3-none-any.whl.
File metadata
- Download URL: tkn-0.1.2-py3-none-any.whl
- Upload date:
- Size: 2.5 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: poetry/1.6.1 CPython/3.11.5 Darwin/22.6.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
e857c6013b413b87f7b1d7a3dc9cbeaa85b5bd2f2633be119ef5cd630f0f9ea5
|
|
| MD5 |
2a5308122a5f287a6a5df03756d4ad2a
|
|
| BLAKE2b-256 |
ccddb26400c68052a803e6c86fab7db3f2031e648ae57560e3608bbb57bef3c5
|
