A Khmer language processing toolkit
Project description
๐ฏTODO
- Sentence Segmentation
- Word Segmentation
- Part of speech Tagging
- Named Entity Recognition
- Text classification
๐ชInstallation
pip install khmer-nltk
๐น Quick tour
To get the evaluation result of khmer-nltk's functionalities, please refer the sub-modules's readme
Sentence tokenization
>>> from khmernltk import sentence_tokenize
>>> raw_text = "แแฝแแแแแถแแแธแขแจ! แขแฃ แแปแแถ แแแแถแแแธแแแแแแแแถแแถแแทแแแถแแแแแแแแทแแแแแแ แแถแแแ
แแแแ
แแแแแแแแแถแ แแถแแแแแแบแแแแแทแแถแ แแทแแแถแแแฝแแแฝแแแถแแแแธ"
>>> print(sentence_tokenize(raw_text))
['แแฝแแแแแถแแแธแขแจ!', 'แขแฃ แแปแแถ แแแแถแแแธแแแแแแแแถแแถแแทแแแถแแแแแแแแทแแแแแแ แแถแแแ
แแแแ
แแแแแแแแแถแ แแถแแแแแแบแแแแแทแแถแ แแทแแแถแแแฝแแแฝแแแถแแแแธ']
Word tokenization
>>> from khmernltk import word_tokenize
>>> raw_text = "แแฝแแแแแถแแแธแขแจ! แขแฃ แแปแแถ แแแแถแแแธแแแแแแแแถแแถแแทแแแถแแแแแแแแทแแแแแแ แแถแแแ
แแแแ
แแแแแแแแแถแ แแถแแแแแแบแแแแแทแแถแ แแทแแแถแแแฝแแแฝแแแถแแแแธ"
>>> print(word_tokenize(raw_text, return_tokens=True))
['แแฝแ', 'แแแแถแ', 'แแธ', 'แขแจ', '!', ' ', 'แขแฃ', ' ', 'แแปแแถ', ' ', 'แแแแถแแแธ', 'แแแแแแแแถ', 'แแถแแท', 'แแแถแ', 'แแแแแ', 'แแทแ', 'แแแแแ', ' ', 'แแถแ', 'แแ
', 'แแแแ
แแ', 'แแแแแแแถแ', ' ', 'แแถแ', 'แแแแแบ', 'แแแแแทแแถแ', ' ', 'แแทแ', 'แแถแแแฝแแแฝแ', 'แแถแแแแธ']
POS Tagging
Usage
>>> from khmernltk import pos_tag
>>> raw_text = "แแฝแแแแแถแแแธแขแจ! แขแฃ แแปแแถ แแแแถแแแธแแแแแแแแถแแถแแทแแแถแแแแแแแแทแแแแแแ แแถแแแ
แแแแ
แแแแแแแแแถแ แแถแแแแแแบแแแแแทแแถแ แแทแแแถแแแฝแแแฝแแแถแแแแธ"
>>> print(pos_tag(raw_text))
[('แแฝแ', 'n'), ('แแแแถแ', 'n'), ('แแธ', 'n'), ('แขแจ', '1'), ('!', '.'), (' ', 'n'), ('แขแฃ', '1'), (' ', 'n'), ('แแปแแถ', 'n'), (' ', 'n'), ('แแแแถแแแธ', 'n'), ('แแแแแแแแถ', 'n'), ('แแถแแท', 'n'), ('แแแถแ', 'o'), ('แแแแแ', 'n'), ('แแทแ', 'o'), ('แแแแแ', 'n'), (' ', 'n'), ('แแถแ', 'v'), ('แแ
', 'v'), ('แแแแ
แแ', 'v'), ('แแแแแแแถแ', 'n'), (' ', 'n'), ('แแถแ', 'v'), ('แแแแแบ', 'n'), ('แแแแแทแแถแ', 'n'), (' ', 'n'), ('แแทแ', 'o'), ('แแถแแแฝแแแฝแ', 'n'), ('แแถแแแแธ', 'o')]
โ๏ธ Citation
@misc{hoang-khmer-nltk,
author = {Phan Viet Hoang},
title = {Khmer Natural Language Processing Tookit},
year = {2020},
publisher = {GitHub},
journal = {GitHub repository},
howpublished = {\url{https://github.com/VietHoang1512/khmer-nltk}}
}
Used in:
- stopes: A library for preparing data for machine translation research
- LASER Language-Agnostic SEntence Representations
- Pretrained Models and Evaluation Data for the Khmer Language
- Multilingual Open Text 1.0: Public Domain News in 44 Languages
- ZusammenQA: Data Augmentation with Specialized Models for Cross-lingual Open-retrieval Question Answering System
- Shared Task on Cross-lingual Open-Retrieval QA
- No Language Left Behind: Scaling Human-Centered Machine Translation
- Wordless
๐จโ๐ References
- NLP: Text Segmentation Using Conditional Random Fields
- Khmer Word Segmentation Using Conditional Random Fields
- Word Segmentation of Khmer Text Using Conditional Random Fields
๐ Advisor
- Prof. Huong Le Thanh
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
khmer-nltk-1.6.tar.gz
(7.0 MB
view hashes)