Skip to main content

Samdei Type

Samdei Type converts romanized Khmer, the Latin spelling many people use in chat, to Khmer script:

$ samdei-type "knhom jong tov phsar"
ខ្ញុំចង់ទៅផ្សារ

This package runs the released Samdei Type models: a command-line converter and a Python API, with beam search, int8 ONNX decoding and the reranking used in the paper. The models are on the Hugging Face Hub and are downloaded on first use:

Model Parameters Decoded as in the paper with First download
Samdei Type Mini 7.1M int8 ONNX graphs + word LM about 9 MB
Samdei Type Base 18.2M PyTorch weights + character LM, word LM and channel model about 125 MB

Both are character-level Transformer encoder-decoders. Mini is small enough for a phone keyboard; Base is more accurate. Their scores on Samdei Type Bench, training data and limitations are in the model cards. All releases: the Samdei Type collection. Paper: technical report (PDF).

Samdei (សំដី, "what you say") is our team, Samdei Labs.

Install

pip install "samdei-type[deploy]"

Python 3.10 or later. The deploy extra adds onnxruntime for the int8 ONNX graphs, which Mini is decoded with. Without it, Mini falls back to its PyTorch weights. PyTorch is needed either way: on macOS the install takes about 1 GB. On Linux, pip installs PyTorch with its CUDA libraries, several GB; for a CPU-only install, add PyTorch's CPU build first: pip install torch --index-url https://download.pytorch.org/whl/cpu.

The latest code from GitHub: pip install "samdei-type[deploy] @ git+https://github.com/Samdei-Labs/samdei-type.git".

Command line

$ samdei-type "knhom jong tov phsar"                      # Samdei Type Mini
ខ្ញុំចង់ទៅផ្សារ
$ samdei-type -m samdei/samdei-type-base "bong nham bay nov?"
បងញ៉ាំបាយនៅ?
$ echo "saek nh tv rean" | samdei-type -                  # one message per line from stdin
ស្អែកខ្ញុំទៅរៀន
$ samdei-type --nbest "jam knhom bontech"                 # also each piece's candidates and scores
ចាំខ្ញុំបន្តិច
       -8.16  ចាំខ្ញុំបន្តិច
      -11.31  ចាំ ខ្ញុំបន្តិច
      ...

samdei-type --help lists the options: --backend torch|onnx, --beam, --no-rerank (beam search only), --no-chunk, --tsv, --device, --threads. python -m samdei_type runs the same command.

The first run downloads the files the model uses into the Hugging Face cache (~/.cache/huggingface). After that it only checks the Hub for updates and works offline; set HF_HUB_OFFLINE=1 to skip the check. -m also takes a local folder:

hf download samdei/samdei-type-mini --local-dir samdei-type-mini
samdei-type -m ./samdei-type-mini "ot mean wifi te"

Python

from samdei_type.pipeline import Converter

conv = Converter("samdei/samdei-type-base")              # a Hub repo id or a local model folder
conv.convert("knhom jong tov phsar")                     # 'ខ្ញុំចង់ទៅផ្សារ'
conv.convert_batch(["bong nham bay nov?", "ot mean wifi te"])
                                                         # ['បងញ៉ាំបាយនៅ?', 'អត់មាន wifi ទេ']
mini = Converter("samdei/samdei-type-mini")              # int8 ONNX graphs, as rerank.json says
mini.nbest(mini.pieces("jam knhom bontech"))             # [[('ចាំខ្ញុំបន្តិច', -8.16), ...]]

Converter reads the model's rerank.json, so it decodes as the published scores were computed: the backend, the beam size, the pieces and the rerank weights. backend="torch" or "onnx" overrides the backend. The same steps with the lower-level pieces:

import json
from huggingface_hub import snapshot_download
from samdei_type.evaluation import chunk_source, translate
from samdei_type.model import load
from samdei_type.rerank import Reranker

d = snapshot_download("samdei/samdei-type-base")
model, sv, tv = load(d)                                  # config.json + model.safetensors
weights = json.load(open(f"{d}/rerank.json"))["rerank_weights"]
rr = Reranker(weights, char_lm=f"{d}/lm/char5_nowiki.npz", word_lm=f"{d}/lm/wordfull_nowiki.json.gz",
              channel=f"{d}/channel")
pieces = chunk_source("knhom jong tov phsar", 8)
nbest = translate(model, sv, tv, pieces, beam=5)
print(" ".join(rr.rerank(p, hyps)[0][0] for p, hyps in zip(pieces, nbest)))

For the int8 graphs, OnnxConverter(f"{d}/onnx").beam(piece, k=5) from samdei_type.onnx_infer gives a piece's n-best list. That module uses only numpy and onnxruntime, and is the reference to port to a phone runtime.

How decoding works

  1. The input is cut into pieces of at most eight words, with a break after any punctuation mark (samdei_type.evaluation.chunk_source). The models were trained on short pieces.
  2. Each piece gets a beam search with k = 5 (samdei_type.model.beam_cached, or OnnxConverter.beam). A piece longer than 249 characters is cut.
  3. Each hypothesis y of input x is scored as log P(y|x) + a·log P_char(y) + b·log P_word(y) − c·#OOV(y) + d·log P_ch(x|y) (samdei_type.rerank). The channel model P_ch reads Khmer and predicts the romanized input, so it checks that a hypothesis explains all of the input. The weights a–d come from the model's rerank.json.
  4. The best hypotheses of the pieces are joined with a space.

Limitations

  • The models convert what people type in chat. They are not a dictionary: an unusual spelling, a name or a rare word can come out as a common word that sounds alike. The model cards give the error rates and examples.
  • English words are often spelled out in Khmer script (see you → ស៊ីយូ); some, like wifi, are kept.
  • Emoji, URLs and @mentions are not passed through: emoji are dropped and other symbols can change. Split them off before converting if you need to keep them.

Package layout

Module What it does
samdei_type.pipeline Converter: a model folder or Hub repo id → convert(text)
samdei_type.model the encoder-decoder, load, KV-cached greedy and beam search
samdei_type.onnx_infer int8 ONNX decoding with numpy and onnxruntime
samdei_type.rerank, samdei_type.lm n-best reranking; hashed character n-gram LM and word bigram LM
samdei_type.evaluation input chunking, batched decoding, CER and exact match (score, as for Samdei Type Bench)
samdei_type.text, samdei_type.khnormal Khmer normalization
samdei_type.vocab character vocabularies
samdei_type.cli the samdei-type command

Training code and the data generator are not part of this package. The training data are on the Hub: samdei/samdei-type-pretraining-data and samdei/samdei-type-finetuning-data.

Development

git clone https://github.com/Samdei-Labs/samdei-type && cd samdei-type
pip install -e ".[deploy]"
python -m unittest discover -s tests
SAMDEI_TYPE_MODEL=samdei/samdei-type-mini python -m unittest discover -s tests   # also an end-to-end conversion

License

The code is licensed under the Apache License 2.0 (see LICENSE and NOTICE). src/samdei_type/khnormal.py is vendored from SIL Global's khmer-character-specification under the MIT License. The models and datasets on the Hub have their own licenses, given in their cards.

Citation

The technical report is paper/samdei-type.pdf.

@misc{samdei-type-2026,
  title  = {Samdei Type: Separating What Is Written from How It Is Spelled for Romanized Khmer Transliteration},
  author = {Mengsay Loem},
  year   = {2026},
  note   = {Technical report},
  url    = {https://github.com/Samdei-Labs/samdei-type/blob/main/paper/samdei-type.pdf}
}

Metadata

Release files for samdei-type 0.1.1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for samdei-type 0.1.1
File Size Uploaded Provenance Actions
samdei_type-0.1.1.tar.gz 28.3 kB With provenance Details for samdei_type-0.1.1.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for samdei-type 0.1.1
File Interpreter ABI Platform Actions
samdei_type-0.1.1-py3-none-any.whl Python 3 none any Details for samdei_type-0.1.1-py3-none-any.whl

Total release size: 59.3 kB

Release files / samdei_type-0.1.1.tar.gz

Download URL samdei_type-0.1.1.tar.gz
Size 28.3 kB
Tags Source
SHA-256 checksum
How to use checksums
2c807c1fe4b88b55d4e849b666f6ff2ea9410f0f77756e0b4c2f35bd2b616dab
BLAKE2b-256 checksum
How to use checksums
931867cf6d4cd20c7f29b1ef51bef7b731ddcc92b109db4ffde3238ae19de6b6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.

Transparency log

Release files / samdei_type-0.1.1-py3-none-any.whl

Download URL samdei_type-0.1.1-py3-none-any.whl
Size 31.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums
0eee9c00d1b00096b68ba1c4a0554bfe6c102b04cb304c470eb4e39461b62550
BLAKE2b-256 checksum
How to use checksums
fa93fc87bf7f7f315c21ca783a85d54cade1d1e6051b1bebedff0d880b94b642
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.16

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.

Transparency log

Security for samdei-type 0.1.1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for samdei_type-0.1.1.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Nov 18, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

0.1.1 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page