A Python Library for Tokenizers
Project description
Lexikanon
A Python Library for Tokenizers
- Documentation: https://lexikanon.entelecheia.ai
- GitHub: https://github.com/entelecheia/lexikanon
- PyPI: https://pypi.org/project/lexikanon
Lexikanon is a robust and efficient Python library designed for creating, training, and deploying tokenizers, an essential component in natural language processing (NLP) and artificial intelligence (AI) applications. The name Lexikanon originates from the Greek words λέξη (word) and κάνων (maker), reflecting the library's purpose in enabling users to build powerful tokenizers for various languages and tasks.
Changelog
See the CHANGELOG for more information.
Contributing
Contributions are welcome! Please see the contributing guidelines for more information.
License
This project is released under the MIT License.
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for lexikanon-0.3.0-py3-none-any.whl
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 | 2413a01e0b176745d536def53a741c2797440718ff38f7d75f10708d8d1328b4 |
|
| MD5 | 112e7192abef2b28fb588fcdfb3dd86f |
|
| BLAKE2b-256 | 26316fdd7e2271659c9f9517f682f8f665e867885016a68d708f11a689164dd0 |
