Thai Tokenizer
Fast and accurate Thai tokenization library using supervised BPE designed for full-text search applications.
Installation
pip3 install thai_tokenizer
Usage
Default set of pairs is optimized for short Thai-English product descriptions.
from thai_tokenizer import Tokenizer
tokenizer = Tokenizer()
tokenizer('iPad Mini 256GB เครื่องไทย') #> 'iPad Mini 256GB เครื่อง ไทย'
tokenizer.split('เครื่องไทย') #> ['เครื่อง', 'ไทย']
Training
See Training for guidelines to train your own pairs.
Contributing
Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.
License
1 maintainer
Metadata
1 maintainer
Release files for thai-tokenizer 0.2.5
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| thai_tokenizer-0.2.5.tar.gz | 6.5 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| thai_tokenizer-0.2.5-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 59.0 kB
Release files / thai_tokenizer-0.2.5.tar.gz
| Download URL | thai_tokenizer-0.2.5.tar.gz |
|---|---|
| Size | 6.5 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
7909d1005a90ce918aa14eb30db47d3b39c9da1106115ccadf3afece53887f24
|
|
BLAKE2b-256 checksum How to use checksums |
80efaf243b5948557d76b51c32588bde7d6f910243241491a85c58a2ea732803
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.1 importlib_metadata/3.10.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.60.0 CPython/3.9.2
|
Release files / thai_tokenizer-0.2.5-py3-none-any.whl
| Download URL | thai_tokenizer-0.2.5-py3-none-any.whl |
|---|---|
| Size | 52.5 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
af7ba74a25e4d444205fc159a2413ca300824d7b3378d7cd6bc26074acb666e4
|
|
BLAKE2b-256 checksum How to use checksums |
812dbb92afb743d152b6ce7c8ea99b232ca615a860130eab2959aedb2ceb4b33
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.4.1 importlib_metadata/3.10.0 pkginfo/1.7.0 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.60.0 CPython/3.9.2
|
