khmercut 0.2.0

khmercut

A (fast) Khmer word segmentation toolkit.

pip install khmercut

Python

from khmercut import tokenize

tokenize("ឃាត់ខ្លួនជនសង្ស័យ០៤នាក់ ករណីលួចខ្សែភ្លើង នៅស្រុកព្រៃនប់")
# => ['ឃាត់ខ្លួន', 'ជនសង្ស័យ', '០៤', 'នាក់', ' ', 'ករណី', 'លួច', 'ខ្សែភ្លើង', ' ', 'នៅ', 'ស្រុក', 'ព្រៃនប់']

Neural model

A distilled neural segmenter. More accurate, and it can split compound words.

pip install khmercut[nn]

from khmercut.nn import tokenize

tokenize("ឃាត់ខ្លួនជនសង្ស័យ០៤នាក់")
# => ['ឃាត់ខ្លួន', 'ជនសង្ស័យ', '០៤', 'នាក់']

tokenize("ឃាត់ខ្លួនជនសង្ស័យ០៤នាក់", deep=True)
# => [['ឃាត់', 'ខ្លួន'], ['ជន', 'សង្ស័យ'], ['០៤'], ['នាក់']]

With deep=True each word becomes a list of its sub-words. Spaces are returned as ' ' and newlines as '\n'.

Both backends load lazily, so importing one does not pull in the other.

Reference

• Khmer language processing toolkit
• Asian Language Treebank