Usage Sample ''''''''''''
.. code:: python
from xtokenizer import Tokenizer
tokenizer = Tokenizer.from_texts(texts, min_freq=5)
sent = 'I love you'
tokens = tokenizer.encode(sent, max_length=6)
# [101, 66, 88, 99, 102, 0]
sent = tokenizer.decode(tokens)
# ['<BOS>', 'I', 'love', 'you', '<EOS>', '<PAD>']
1 maintainer
Release files for xtokenizer 0.0.6
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| xtokenizer-0.0.6.tar.gz | 18.9 kB | Details |
Release files / xtokenizer-0.0.6.tar.gz
| Download URL | xtokenizer-0.0.6.tar.gz |
|---|---|
| Size | 18.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
719e028ce65cb17fdce8a843d41820e8f04d167457c160f0dc8f8fd3a139e7f2
|
|
BLAKE2b-256 checksum How to use checksums |
132094505706b015ae89398220b24bf720e0b665d754ff19bcb7609bd2fb7b84
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.0.0 CPython/3.9.18
|
