CLAWS Tokenizer: Cluster Level Aggregation Khmer Word Segmentation

Claws is a lightweight, fast Khmer word segmentation and tokenization library for Python powered by LiteRT (formerly TensorFlow Lite runtime).

Features

• Accurate Khmer Segmentation: Handles zero-width spaces, compound words, and standard continuous Khmer script.
• Lightweight & Fast: Uses LiteRT for fast inference with minimal runtime overhead.
• Zero Heavy Dependencies: Requires only numpy and ai-edge-litert.
• Pre-packaged Model: Pre-trained model and vocabulary files are bundled directly inside the wheel.

Installation

Install via pip:

pip install claws-tokenizer

Quick Start

from claws_tokenizer import segment

tokens = segment("ខ្ញុំស្រឡាញ់ភាសាខ្មែរ")
print(tokens)
# Output: ['ខ្ញុំ', 'ស្រឡាញ់', 'ភាសា', 'ខ្មែរ']

Citation

If you use Claws in your research, please cite it as follows:

BibTeX

@software{claws_tokenizer,
  author = {Socret Lee},
  title = {CLAWS Tokenizer: Cluster Level Aggregation Khmer Word Segmentation},
  year = {2026},
  url = {[https://github.com/Socret360/claws](https://github.com/Socret360/claws)},
  version = {0.1.0}
}