Byte Pair Encoding Tokenizer with special tokens and regex pattern
Project description
bpetokenizer
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
bpetokenizer-1.0.0.tar.gz
(1.8 kB
view hashes)
Built Distribution
Close
Hashes for bpetokenizer-1.0.0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0a6cd9db2f86d3d20fe7431f22c5183364220cc36f9fb13ef645e143d3993afc |
|
MD5 | 0023538405754d9b7da33970df30945b |
|
BLAKE2b-256 | 42d64ac3bec2e93b470c0e332d08112f8b085be52a1ae677c417e70f9c84b70a |