CocCocTokenizer deployed by trituenhantao.io.
Project description
This project provides tokenizer library for Vietnamese language and 2 command line tools for tokenization and some simple Vietnamese-specific operations with text (i.e. remove diacritics). It is used in Cốc Cốc Search and Ads systems and the main goal in its development was to reach high performance while keeping the quality reasonable for search ranking needs.
Deployed by trituenhantao.io.
Installing
pip install cython
pip install CocCocTokenizer
Using Python bindings
from CocCocTokenizer import PyTokenizer
# load_nontone_data is True by default
T = PyTokenizer(load_nontone_data=True)
# tokenize_option:
# 0: TOKENIZE_NORMAL (default)
# 1: TOKENIZE_HOST
# 2: TOKENIZE_URL
print(T.word_tokenize("xin chào, tôi là người Việt Nam", tokenize_option=0))
# output: ['xin', 'chào', ',', 'tôi', 'là', 'người', 'Việt_Nam']
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Filter files by name, interpreter, ABI, and platform.
If you're not sure about the file name format, learn more about wheel file names.
Copy a direct link to the current filters
File details
Details for the file CocCocTokenizer-0.1.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.
File metadata
- Download URL: CocCocTokenizer-0.1.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- Upload date:
- Size: 195.2 MB
- Tags: CPython 3.8, manylinux: glibc 2.17+ x86-64
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/4.0.1 CPython/3.8.3
File hashes
| Algorithm | Hash digest | |
|---|---|---|
| SHA256 |
1749dad69d30ebbef6f8620054165c9741f11dbc032dbc1a3ed9ab47f685dfbe
|
|
| MD5 |
596ef7d65cf12d4ba1ecfc67346d76f8
|
|
| BLAKE2b-256 |
52258d0fffa4768540053fb6b2bfe882742e08ee26b354d6d38e8a79beeda676
|
