Khmer Segmenter

A lightweight Khmer word segmentation package for Python ≥ 3.10, designed for simple and efficient tokenization.

This project is adapted and simplified from the original khnlp package, with the following goals:

• Support modern Python versions (≥ 3.10)
• Simplified installation
• Focus on a single task: Khmer word segmentation
• Lightweight and easy to integrate into NLP pipelines

This package is intended as a small academic contribution to support Khmer NLP research and practical applications.

Installation

pip install khmer-segmenter

Requires:

• Python >= 3.10

Usage

from khmer_segmenter import Tokenizer

tokenizer = Tokenizer(seg_type="com")

print(tokenizer.tokenize("សួស្ដីអ្នកទាំងអស់យើង"))

Segmentation Modes

The tokenizer supports two segmentation strategies:

1. Compound-based (seg_type="com")

• Segments text into compound words
• Suitable for general word-level NLP tasks
• Recommended for downstream applications such as:
  • Text classification
  • Named Entity Recognition
  • Information retrieval

2. Morpheme-based (seg_type="mor")

• Performs finer-grained segmentation
• Splits text into smaller morphological units
• Useful for:
  • Linguistic analysis
  • Subword modeling
  • Research-focused NLP tasks

Motivation

Khmer is a low-resource language with no explicit word boundary markers (spaces are not consistently used to separate words). This creates challenges for:

• Automatic Speech Recognition (ASR)
• Language Modeling
• Machine Translation
• Information Extraction

Existing tools for Khmer segmentation often have:

• Limited Python version support
• Heavy dependencies
• Broader NLP scope than necessary

This project provides a focused, minimal, and modern alternative dedicated solely to segmentation.