Pheasa (ភាសា)

Open reference infrastructure for Khmer language AI. Pheasa is the romanization of ភាសា, the Khmer word for "language".

Pheasa aims to provide:

• a normalizer, so that Khmer text that looks identical is also byte-identical;
• a benchmark for measuring how well language models handle Khmer;
• datasets and baselines with documented provenance.

Status: alpha. Version 0.1.0 ships the normalizer (normalization version 1). The benchmark and datasets are in progress.

The problem

Unicode allows several code point orders for the same visible Khmer syllable. Fonts render them alike, but search, deduplication, tokenizers, and evaluation metrics treat them as different strings:

Unicode NFC does not reconcile these. Pheasa's normalizer does, following the Khmer encoding structure described in Unicode Technical Note #61.

from pheasa import normalize

assert normalize("ខែ្មរ") == normalize("ខ្មែរ") == "ខ្មែរ"

normalize(text, report=True) also returns every change it made, with input offsets and the rule responsible, plus any problems it could not safely fix. Options that remove or rewrite information (zwsp, digits, fold_deprecated) are off by default.

From the command line:

pheasa normalize input.txt -o output.txt --report changes.jsonl
pheasa validate input.txt

See the user guide for options, the report format and issue codes. The rules are specified in spec/normalization.md.

Install

pip install pheasa

Development

Requires uv.

uv sync
make check

Acknowledgements

Pheasa builds on the Khmer encoding structure work by SIL International (khmer-character-specification) and Unicode Technical Note #61.

Citation

If you use Pheasa, please cite it using CITATION.cff.

License

Code is licensed under Apache-2.0. Datasets are released under CC BY 4.0 unless their sources require stricter terms.

Created and maintained by Samputhy Khim (@organi-cs, ORCID 0009-0008-4510-8839).