newmm-tokenizer
Standalone Dictionary-based, Maximum Matching + Thai Character Cluster (newmm) tokenizer extracted from PyThaiNLP.
Objectives
This repository is created for reducing an overall size of original PyThaiNLP Tokenizer Module. The main objective is to be able to segment Thai sentences into a list of words.
Supports
The module supports Python 3.7+ as follow the original PyThaiNLP repository.
Installation
pip install newmm-tokenizer
How to Use
from newmm_tokenizer.tokenizer import word_tokenize
text = 'เป็นเรื่องแรกที่ร้องไห้ตั้งแต่ ep 1 แล้วก็เป็นเรื่องแรกที่เลือกไม่ได้ว่าจะเชียร์พระเอกหรือพระรองดี 19...'
words = word_tokenize(text)
print(words)
# ['เป็นเรื่อง', 'แรก', 'ที่', 'ร้องไห้', 'ตั้งแต่', ' ', 'ep', ' ', '1', ' ', 'แล้วก็', 'เป็นเรื่อง', 'แรก', 'ที่', 'เลือกไม่ได้', 'ว่า', 'จะ', 'เชียร์', 'พระเอก', 'หรือ', 'พระรอง', 'ดี', ' ', '19', '...']
LICENSE
Please see the original license of PyThaiNLP here
Release files for newmm-tokenizer 0.2.2
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| newmm_tokenizer-0.2.2.tar.gz | 314.2 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| newmm_tokenizer-0.2.2-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 634.8 kB
Release files / newmm_tokenizer-0.2.2.tar.gz
| Download URL | newmm_tokenizer-0.2.2.tar.gz |
|---|---|
| Size | 314.2 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
e6bd825d6a05f759be1e9be67e1d603a61f961d3fa9979d4c3af21ae576250ec
|
|
BLAKE2b-256 checksum How to use checksums |
aca80135c90ddeaae26f1e12cfd08f4a55e4ff6b2edd5228d7eaba7a55c90f0b
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.7.1 importlib_metadata/4.10.0 pkginfo/1.8.2 requests/2.27.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.7
|
Release files / newmm_tokenizer-0.2.2-py3-none-any.whl
| Download URL | newmm_tokenizer-0.2.2-py3-none-any.whl |
|---|---|
| Size | 320.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
2ef54c67585d0f562650c93368a07ba0e39b5c8dc4500991135c57df58da65a5
|
|
BLAKE2b-256 checksum How to use checksums |
96f2e93d15afba1dec377d3a4c018ec1f75214510d6a1792ea9ecee526f5089d
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/3.7.1 importlib_metadata/4.10.0 pkginfo/1.8.2 requests/2.27.1 requests-toolbelt/0.9.1 tqdm/4.62.3 CPython/3.8.7
|
