Иерархический TOV-токенизатор с γ-кэшированием и λ-утечкой
Project description
TOV Tokenizer
Иерархический токенизатор на основе Теории Относительности Всего (ТОВ).
Особенности
- 🌳 Иерархическое дерево (предложения → слова → подслова → символы)
- ⚡ γ-кэширование частых токенов
- 🧹 λ-утечка для редких токенов
- 🧠 Выбор токенов через потоки массы
- 🔄 Сохранение/загрузка состояния
Установка
pip install tov-tokenizer
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
tov_tokenizer-0.1.0.tar.gz
(6.9 kB
view details)
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file tov_tokenizer-0.1.0.tar.gz.
File metadata
- Download URL: tov_tokenizer-0.1.0.tar.gz
- Upload date:
- Size: 6.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.14.3
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
0d8046822512d16f76facf10f6690a6ae2d57ae578cbe2054ef8dd6db2b988c0
|
|
| MD5 |
ee4e79ec1e7355ff22a7659ea7f4f6c7
|
|
| BLAKE2b-256 |
0a46d8d10bf44bf5ec70f71cf838ea2b93c7c07069e8fe35508985b3b24b2da4
|
File details
Details for the file tov_tokenizer-0.1.0-py3-none-any.whl.
File metadata
- Download URL: tov_tokenizer-0.1.0-py3-none-any.whl
- Upload date:
- Size: 7.9 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/6.2.0 CPython/3.14.3
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
c651a5aaa62a029aa6e08490153e055ce39b1a375f119dbcdecc79fded4da9e5
|
|
| MD5 |
1236f817d594596485899c0e526e4727
|
|
| BLAKE2b-256 |
6070d331f3fcbd9caf27b86353f7fddb98d5b9ac0425bef05434a16e545d88bb
|
