CocCocTokenizer deployed by trituenhantao.io.
Project description
This project provides tokenizer library for Vietnamese language and 2 command line tools for tokenization and some simple Vietnamese-specific operations with text (i.e. remove diacritics). It is used in Cốc Cốc Search and Ads systems and the main goal in its development was to reach high performance while keeping the quality reasonable for search ranking needs.
Deployed by trituenhantao.io.
Installing
pip install cython
pip install CocCocTokenizer
Using Python bindings
from CocCocTokenizer import PyTokenizer
# load_nontone_data is True by default
T = PyTokenizer(load_nontone_data=True)
# tokenize_option:
# 0: TOKENIZE_NORMAL (default)
# 1: TOKENIZE_HOST
# 2: TOKENIZE_URL
print(T.word_tokenize("xin chào, tôi là người Việt Nam", tokenize_option=0))
# output: ['xin', 'chào', ',', 'tôi', 'là', 'người', 'Việt_Nam']
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
No source distribution files available for this release.See tutorial on generating distribution archives.
Built Distribution
Close
Hashes for CocCocTokenizer-0.1.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 1749dad69d30ebbef6f8620054165c9741f11dbc032dbc1a3ed9ab47f685dfbe |
|
MD5 | 596ef7d65cf12d4ba1ecfc67346d76f8 |
|
BLAKE2b-256 | 52258d0fffa4768540053fb6b2bfe882742e08ee26b354d6d38e8a79beeda676 |