A tiny tokenizer library for tiny Generative Pretrained Transformers.
Project description
EzTokenizer
EzTokenizer is a simple and user-friendly tokenizer for small language models and/or Generative Pretrained Transformers (GPT's).
It's main purpose is to automate and make tokenization easy, so you don't have to program one from scratch.
Inspirations
EzTokenizer was inspired by Transformers's AutoTokenizer feature.
What makes EzTokenizer stand out is that it automates tokenization, is easy to use, and it doesn't rely on a pre-trained vocabulary.
About the usage of AI
AI was used for instructions on how to get started, all of EzTokenizer's code is hand-written.
Thank you for reading! - im-lemon
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file eztokenizer-0.1.3.tar.gz.
File metadata
- Download URL: eztokenizer-0.1.3.tar.gz
- Upload date:
- Size: 1.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.14.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
52e10f8164727c2e39da24b0cff3dfd7b1a2918a3277bbb00ead1bde8f9be76c
|
|
| MD5 |
a8bca18205fa6fa3f50f027af1dd6655
|
|
| BLAKE2b-256 |
7b24328f7d2591d51272385ed3a3c2c4b40a1d8c27563106523d14aaa5c4d5bc
|
File details
Details for the file eztokenizer-0.1.3-py3-none-any.whl.
File metadata
- Download URL: eztokenizer-0.1.3-py3-none-any.whl
- Upload date:
- Size: 2.2 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.14.0
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
9572a0b288ec051c7a4c38a621d120d42669d5e75afa6998585c5a32234686ae
|
|
| MD5 |
e28c7165a4d8977b13c916259f82baec
|
|
| BLAKE2b-256 |
7895599ba3835c9bd8f24381f86b3e6dec9beb4dc9911c6aced04d9729711b22
|
