A missing toolkit for Khmer Natural Language Processing.
Project description
Khmer Normalizer
A missing toolkit for Khmer Natural Language Processing.
- Character Reordering
- Duplicate Whitespaces
- Remove zero width space
- Remove emojis
- Fix Common misspellings
- Fix Unicode issues
- Fix Khmer trailing vowels
- URL Replacements
- Unicode Normalization (NFKC)
- Quotes symbols normalization
- Remove repeated punctuations
Installation
pip install khmernormalizer
Usage
from khmernormalizer import normalize
input_str = """
តាម៖៖សេចក្តីរាយការណ៍ឲ្យដឹងថា!!!!!
https://google.com/a?x=1
កាល 😂 ពីវេលាម៉ោង ៗ ប្រមាណ១១យប់ថ្ងៃទី៤ 😂😂😂😂😂 ??
កាាាាត់
មិិិិិន
មួយរយះះះះះះះ
រយះពេល
""".strip()
normalize(input_str,
emoji_replacement="",
remove_zwsp=True,
url_replacement="")
Result:
តាម៖សេចក្តីរាយការណ៍ឱ្យដឹងថា!
កាល ពីវេលាម៉ោងៗ ប្រមាណ១១យប់ថ្ងៃទី៤?
កាត់
មិន
មួយរយៈះ
រយៈពេល
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
khmernormalizer-0.0.3.tar.gz
(6.7 kB
view hashes)
Built Distribution
Close
Hashes for khmernormalizer-0.0.3-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | eac4bcd3a0dd70f7cd2622a7f5f2065775aa1dd2975726cbf4d1ab8e74aee294 |
|
MD5 | fd39d7089bbb588bb68ab37da1e8a45d |
|
BLAKE2b-256 | 2466e2cb96ae9a105c9ff2b0f40d025af4e248e1678380cf1addfeb9a413d82c |