khmer-nlp-toolkits 0.2.1

Khmer NLP toolkits

khmer_nlp_toolkits

Khmer NLP Toolkits is an open-source Python library that provides practical utilities, text processing functions, and NLP components for the Khmer language.

The project is designed to make Khmer NLP easier for:

• Researchers
• Students
• Developers
• Production systems
• AI applications

It aims to provide a unified, lightweight, and extensible toolkit for common Khmer language processing tasks.

📖 Documentation | 💻 GitHub | 📦 PyPI

Installation

Requirements

• Python 3.10+

Install from PyPI

pip install khmer_nlp_toolkits

Quick Start

from khmer_nlp_toolkits.text import *
from khmer_nlp_toolkits.number import num2text

print(num2text(2026))
# ពីរពាន់ម្ភៃប្រាំមួយ

print(anonymizer(("You can contact us by phone at [phone], email us at [e-mail], "
"or visit our website at https://www.example.com anytime.")))
# You can contact us by phone at [TEL], email us at [EML], or visit our website at [URL] anytime.

print(text_cleaner(
    "\u200b Hello\u2014\u2014World!!!??      ខ្មែរ   123\u200b "
))
# "Hello - World !? ខ្មែរ 123"

Features

Khmer NLP Toolkits provides modular utilities and NLP components for Khmer language processing.

Commoncrawl 🔗

• Content Quality Warning
• Document Quality Filtering
• Adult Content Filtering

Text Processing 🔗

• Text Anonymization
• Text Normalization
• Text Cleaning
• Segmentation

Number Processing 🔗

• number conversion to text

Deduplication 🔗

• URL deduplication
• Content Deduplication

Utilities 🔗

• End2End Pipeline
• File Utilities Features

Vision

The long-term goal of Khmer NLP Toolkits is to help build a comprehensive open-source ecosystem for Khmer Natural Language Processing and accelerate Khmer AI/NLP research and development.

We hope this project can contribute to:

• Better Khmer language tooling
• Educational resources
• NLP research accessibility
• Open-source AI infrastructure for Khmer

Contributing

Contributions, issues, and feature requests are welcome. Feel free to open a pull request or discussion.

Planned Features

• Khmerize
• NER
• Semantic Similarity
• Text2Num
• etc.

Contributors

• Sopagna HEANG
• Natt KORAT
• Chily RAN

License

This project is licensed under the MIT License.

References

[1] Hour Kaing, Raj Dabre, Haiyue Song, Van-Hien Tran, Hideki Tanaka, and Masao Utiyama. 2025. PrahokBART: A Pre-trained Sequence-to-Sequence Model for Khmer Natural Language Generation. In Proceedings of the 31st International Conference on Computational Linguistics, pages 1309–1322, Abu Dhabi, UAE. Association for Computational Linguistics.

[2] Martin Hosken, Norbert Lindenberg, and Makara Sok. 2022. Khmer encoding structure. Technical report, The Unicode Technical Committee.

[3] Hoang, P. V. (2020). Khmer Natural Language Processing Toolkit. GitHub repository. https://github.com/VietHoang1512/khmer-nltk

[4] Vichet Chea, Ye Kyaw Thu, Chenchen Ding, Masao Utiyama, Andrew Finch, and Eiichiro Sumita. Khmer word segmentation using conditional random fields. Khmer Natural Language Processing, 2015