Khmer Normalizer
A missing toolkit for Khmer Natural Language Processing.
- Character Reordering
- Duplicate Whitespaces
- Remove zero width space
- Remove emojis
- Fix Common misspellings
- Fix Unicode issues
- Fix Khmer trailing vowels
- URL Replacements
- Unicode Normalization (NFKC)
- Quotes symbols normalization
- Remove repeated punctuations
Installation
pip install khmernormalizer
Usage
from khmernormalizer import normalize
input_str = """
តាម៖៖សេចក្តីរាយការណ៍ឲ្យដឹងថា!!!!!
https://google.com/a?x=1
កាល 😂 ពីវេលាម៉ោង ៗ ប្រមាណ១១យប់ថ្ងៃទី៤ 😂😂😂😂😂 ??
កាាាាត់
មិិិិិន
មួយរយះះះះះះះ
រយះពេល
""".strip()
normalize(input_str,
emoji_replacement="",
remove_zwsp=True,
url_replacement="")
Result:
តាម៖សេចក្តីរាយការណ៍ឱ្យដឹងថា!
កាល ពីវេលាម៉ោងៗ ប្រមាណ១១យប់ថ្ងៃទី៤?
កាត់
មិន
មួយរយៈ
រយៈពេល
1 maintainer
Metadata
1 maintainer
Release files for khmernormalizer 0.0.4
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| khmernormalizer-0.0.4.tar.gz | 10.0 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| khmernormalizer-0.0.4-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 19.9 kB
Release files / khmernormalizer-0.0.4.tar.gz
| Download URL | khmernormalizer-0.0.4.tar.gz |
|---|---|
| Size | 10.0 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
f39925b0ea420b1a27f3d6bd3add973739ebc73300bbc4893c7e703a38b4d5e8
|
|
BLAKE2b-256 checksum How to use checksums |
ec73277eae471aad7ea0927d5b26fd482e49b6109942c0ff784be76bda204ef4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.10.0
|
Release files / khmernormalizer-0.0.4-py3-none-any.whl
| Download URL | khmernormalizer-0.0.4-py3-none-any.whl |
|---|---|
| Size | 9.9 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
86c0cfd1021d13b754f0d8c47b1cff950b6a16152f862a09548abd88f3eb97b0
|
|
BLAKE2b-256 checksum How to use checksums |
b4506549f7c5758f70bf6ea1a2c8160a2137de12ec1b0b9446abe116ff59c5bf
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.2 CPython/3.10.0
|
