Usage Sample ''''''''''''
.. code:: python
from xtokenizer import Tokenizer
tokenizer = Tokenizer.from_texts(texts, min_freq=5)
sent = 'I love you'
tokens = tokenizer.encode(sent, max_length=6)
# [101, 66, 88, 99, 102, 0]
sent = tokenizer.decode(tokens)
# ['<BOS>', 'I', 'love', 'you', '<EOS>', '<PAD>']
1 maintainer
Release files for xtokenizer 0.0.5
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| xtokenizer-0.0.5.tar.gz | 18.9 kB | Details |
Release files / xtokenizer-0.0.5.tar.gz
| Download URL | xtokenizer-0.0.5.tar.gz |
|---|---|
| Size | 18.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
1e8fdce129a30ebfa50e816c568478182b47bf744f5e3d0972484e79542ee68f
|
|
BLAKE2b-256 checksum How to use checksums |
d464c813f68100d3ab7ff9c7a3db5bd9258524ac8d9327733c54a1622e71bfd2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.0.0 CPython/3.9.18
|
