LEKCut
LEKCut (เล็ก คัด) is a Thai tokenization library that ports the deep learning model to the onnx model.
Install
pip install lekcut
How to use
from lekcut import word_tokenize
# DeepCut model (default)
word_tokenize("ทดสอบการตัดคำ")
# output: ['ทดสอบ', 'การ', 'ตัด', 'คำ']
# AttaCut syllable + character model
word_tokenize("ทดสอบการตัดคำ", model="attacut-sc")
# output: ['ทดสอบ', 'การ', 'ตัด', 'คำ']
# AttaCut character-only model
word_tokenize("ทดสอบการตัดคำ", model="attacut-c")
# output: ['ทดสอบ', 'การ', 'ตัด', 'คำ']
# OSKut model
word_tokenize("เบียร์ยูไม่อร่อย", model="oskut")
# output: ['เบียร์', 'ยู', 'ไม่', 'อ', 'ร่อย']
# OSKut with a specific engine
word_tokenize("เบียร์ยูไม่อร่อย", model="oskut", engine="tnhc")
# output: ['เบียร์', 'ยู', 'ไม่', 'อร่อย']
# SEFR_CUT model
word_tokenize("เบียร์ยูไม่อร่อย", model="sefr-tnhc")
# output: ['เบียร์', 'ยู', 'ไม่', 'อร่อย']
API
word_tokenize(
text: str,
model: str = "deepcut",
path: str = "default",
providers: List[str] = None,
engine: str = "ws",
k: int = 1,
) -> List[str]
Parameters:
text: Text to tokenizemodel: Model to use. Options:"deepcut"(default),"attacut-sc","attacut-c","oskut","sefr-best","sefr-tnhc","sefr-ws1000"path: Path to custom model file (default: "default", applies todeepcutandattacut-*models)providers: List of ONNX Runtime execution providers (default: None, which uses default CPU provider)engine: OSKut engine variant (applies to"oskut"model only). Options:"ws"(default),"ws-augment-60p","tnhc","scads","tl-deepcut-ws","tl-deepcut-tnhc","deepcut"k: Percentage of characters to refine for OSKut (applies to"oskut"model only). The special default value of1is a sentinel that lets OSKut automatically select an appropriate percentage based on the engine. Pass any integer from 2 to 100 to override.
GPU Support
LEKCut supports GPU acceleration through ONNX Runtime execution providers. To use GPU acceleration:
-
Install ONNX Runtime with GPU support:
pip install onnxruntime-gpu
-
Use the
providersparameter to specify GPU execution:from lekcut import word_tokenize # Use CUDA GPU result = word_tokenize("ทดสอบการตัดคำ", providers=['CUDAExecutionProvider', 'CPUExecutionProvider']) # Use TensorRT (if available) result = word_tokenize("ทดสอบการตัดคำ", providers=['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider'])
Available Execution Providers:
CPUExecutionProvider- Default CPU executionCUDAExecutionProvider- NVIDIA CUDA GPU accelerationTensorrtExecutionProvider- NVIDIA TensorRT optimizationDmlExecutionProvider- DirectML for Windows GPU- And more (see ONNX Runtime documentation)
Note: The providers are tried in order, and the first available one will be used. Always include CPUExecutionProvider as a fallback.
Model
deepcut- We ported deepcut model from tensorflow.keras to ONNX model. The model and code come from Deepcut's Github. The model is here.attacut-sc- We ported the AttaCut syllable + character model from PyTorch to ONNX. The model and code come from AttaCut's Github. Requires thessgpackage for syllable tokenization.attacut-c- We ported the AttaCut character-only model from PyTorch to ONNX. The model and code come from AttaCut's Github.oskut- We ported the OSKut (Out-of-domain Stacked Cut) stacked ensemble models from TensorFlow/Keras to ONNX. The model and code come from OSKut's Github. Requires thepyahocorasickpackage. Supports multiple engines:ws(default),ws-augment-60p,tnhc,scads,tl-deepcut-ws,tl-deepcut-tnhc,deepcut.SEFR_CUT- We ported the SEFR CUT (Stacked Ensemble Filter and Refine for Word Segmentation) model from PyTorch to ONNX. The model and code come from SEFR_CUT's Github. List models:"sefr-best","sefr-tnhc","sefr-ws1000"
Load custom model
If you have trained your custom model from deepcut or other that LEKCut support, You can load the custom model by path in word_tokenize after porting your model.
- How to train custom model with your dataset by deepcut - Notebook (Needs to update
deepcut/train.pybefore train model)
How to porting model?
See notebooks/
Metadata
Release files for LEKCut 1.0.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | Provenance | Actions |
|---|---|---|---|---|
| lekcut-1.0.0.tar.gz | 24.9 MB | Without provenance | Details for lekcut-1.0.0.tar.gz |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| lekcut-1.0.0-py3-none-any.whl | Python 3 | none | any | Details for lekcut-1.0.0-py3-none-any.whl |
Total release size: 49.8 MB
Release files / lekcut-1.0.0.tar.gz
| Download URL | lekcut-1.0.0.tar.gz |
|---|---|
| Size | 24.9 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
5156330d6ce36812dd9fa710bc43918a3cdbff3823cef822e61bbad7728de589
|
|
BLAKE2b-256 checksum How to use checksums |
3142d5fc35d4babf2e0354c7bf74acca840a1657c38c55edc6a054041d3eb5a1
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Release files / lekcut-1.0.0-py3-none-any.whl
| Download URL | lekcut-1.0.0-py3-none-any.whl |
|---|---|
| Size | 24.9 MB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
f9b0be4bcb01215f6615efb0d44f48b96f27435264d1d45240985fe2a085201e
|
|
BLAKE2b-256 checksum How to use checksums |
d3415117f08cc9c9d7e83174877d509f76610eea8164dd999dfc574dedcdff81
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.13.7
|
Security for LEKCut 1.0.0
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.