orthomni-codeco

Vietnamese spelling correction on CPU with ONNX Runtime INT8, no PyTorch required. The model is a Seq2Seq BARTpho/MBart (~400M parameters).

• Lightweight: the only dependencies are onnxruntime, numpy, sentencepiece and huggingface-hub (~120 MB, versus ~1.3 GB with torch + transformers).
• Automatic model download: the wheel contains no weights. On first run, the INT8 model (~1.1 GB) is downloaded from the Hugging Face Hub to ~/.cache/orthomni_codeco/ with a progress bar. Later runs work offline.
• Batteries included: FastAPI server, CLI and Dockerfile.

Installation

pip install orthomni-codeco            # library
pip install "orthomni-codeco[server]"  # + FastAPI/uvicorn

Requires Python >= 3.8.

Quickstart

from orthomni_codeco import OrthoMNIModel

model = OrthoMNIModel()                         # first run: downloads the model to the cache, then loads it (~2 s)
model.correct("hom nay troi dep qua")           # 'Hôm nay trời đẹp quá.'
model.correct(["xin chao", "toi di hoc"])       # ['Xin chào.', 'Tôi đi học.']

predict() and generate() are aliases of correct(). Corrector is the old name of OrthoMNIModel.

Library usage

from orthomni_codeco import OrthoMNIModel, CorrectorConfig, InputTooLongError, get_corrector, ensure_model

model = get_corrector()                                   # per-process singleton, config read from ORTHOMNI_* env vars
model = OrthoMNIModel(CorrectorConfig(intra_op_threads=4))
model = OrthoMNIModel(CorrectorConfig(model_dir="/path/to/int8"))  # use a local model, no download

await model.acorrect("xin chao")                          # async, does not block the event loop

try:
    model.correct(very_long_text)
except InputTooLongError as e:
    print(e.indices, e.max_tokens)                        # which inputs exceed the limit -> split sentences first

ensure_model()                                            # only download the model to the cache, returns its path

• Load once: create OrthoMNIModel once and reuse it.
• Thread-safe: calls are serialized with a lock, since ONNX Runtime already uses all the threads it is given.
• Lightweight import: import orthomni_codeco does not load onnxruntime or numpy.

Model source

In order of priority:

1. model_dir / ORTHOMNI_MODEL_DIR: a local model directory, nothing is downloaded.
2. repo_id + revision (ORTHOMNI_REPO_ID, ORTHOMNI_REVISION): downloaded from the Hugging Face Hub to cache_dir (ORTHOMNI_CACHE_DIR, default ~/.cache/orthomni_codeco/<org>--<repo>/<revision>/).

Private repos use HF_TOKEN or hf auth login. For offline environments, pre-download with orthomni download, then set HF_HUB_OFFLINE=1.

Mounting into an existing FastAPI app

import asyncio
from contextlib import asynccontextmanager
from fastapi import FastAPI
from orthomni_codeco import get_corrector
from orthomni_codeco.server import create_router

@asynccontextmanager
async def lifespan(app: FastAPI):
    await asyncio.to_thread(get_corrector)   # load the model at startup
    yield

app = FastAPI(lifespan=lifespan)
app.include_router(create_router(), prefix="/spell")   # -> POST /spell/predict, GET /spell/health

CLI and standalone server

orthomni download                                   # download the model to the cache, print its path
orthomni correct "toi di hoc" "xin chao"            # quick test
orthomni serve --port 8006                          # workers/threads derived from the vCPU count
orthomni serve --workers 1 --threads 8              # lowest latency

python -m orthomni_codeco ... is equivalent to orthomni ....

Method	Path	Description
GET	/health	200 {"status":"ok"} when the model is ready, 503 otherwise
POST	/predict	Body {"text": str} or {"texts": list[str]}

curl -X POST localhost:8006/predict -H 'content-type: application/json' -d '{"text":"hom nay troi dep qua"}'
# {"result":"Hôm nay trời đẹp quá.","latency_ms":113.7}

Error codes: 422 invalid input (both text and texts, batch too large, empty text), 413 sentence longer than MAX_INPUT_TOKENS (the server does not truncate), 500 inference error.

Configuration (environment variables)

Variable	Default	Meaning
ORTHOMNI_MODEL_DIR	(empty)	Local model directory; when set, no download happens
ORTHOMNI_REPO_ID / ORTHOMNI_REVISION	see config.py	Hugging Face Hub repo and revision
ORTHOMNI_CACHE_DIR	~/.cache/orthomni_codeco	Model cache directory
ORTHOMNI_INTRA_OP_THREADS	0 (= vCPU / workers)	ORT threads within one operator. Biggest effect on latency
ORTHOMNI_INTER_OP_THREADS	1	Parallelism across graph branches
WEB_CONCURRENCY	vCPU / 4	Number of uvicorn workers; each holds its own model copy (~1.7 GB RAM)
ORTHOMNI_MAX_INPUT_TOKENS	256	Maximum tokens per sentence
ORTHOMNI_INFER_BATCH_SIZE	8	Sub-batch size (sorted by length to reduce padding)
ORTHOMNI_NEW_TOKENS_RATIO / _OFFSET / _CAP	1.3 / 8 / 320	max_new_tokens = min(CAP, len_input × RATIO + OFFSET)
ORTHOMNI_WARMUP	true	Run one dummy inference on load
ORTHOMNI_MAX_BATCH_SIZE	32	Maximum number of texts per request
ORTHOMNI_MAX_TEXT_CHARS	4000	Maximum characters per text
ORTHOMNI_MAX_CONCURRENCY	1	Concurrent generate requests per worker

Avoid thread thrashing: keep workers × ORTHOMNI_INTRA_OP_THREADS ≤ vCPUs. The serve command reads CPU limits from cgroups (docker --cpus, K8s limits). Fewer workers with more threads gives lower latency; more workers with fewer threads gives higher throughput.

Docker

Requires the INT8 model in ./model (see below):

docker build -t orthomni-codeco:cpu .
docker run --rm -p 8006:8006 --cpus 8 --memory 4g orthomni-codeco:cpu

Development

Environment setup

python -m venv .venv && . .venv/bin/activate
pip install -e ".[server,dev]"
ORTHOMNI_MODEL_DIR=./model pytest -q        # model tests are skipped if ./model is missing
./run.sh                                    # local server using ./model, port 7428

Exporting a new model

The current INT8 model is quantized for arm64 (see model_info.json). It still runs on x86 but not optimally, so re-export on a machine with the same CPU architecture as the serving machine:

pip install torch --index-url https://download.pytorch.org/whl/cpu   # CPU-only torch first
pip install -e ".[export]"
orthomni export --model_dir /path/to/checkpoint --output_dir ./model [--target avx512_vnni]

Quantization is dynamic INT8 per-channel, applied only to MatMul; embeddings stay FP32. Per-tensor quantization produced incorrect sentences.

Publishing the model to the Hugging Face Hub

hf auth login
python scripts/upload_model.py --model_dir ./model --repo_id <org>/orthomni-codeco

The script prints a commit hash. Update DEFAULT_REPO_ID and DEFAULT_REVISION in src/orthomni_codeco/config.py before releasing, so each package version always downloads the same set of weights.

Build and publish

pip install -U build twine
rm -rf dist/ && python -m build                  # -> dist/*.whl + dist/*.tar.gz
twine check dist/*                               # check metadata and that the README renders on PyPI
unzip -l dist/*.whl                              # confirm no .onnx files are inside

# Try the wheel in a clean venv
python -m venv /tmp/t && /tmp/t/bin/pip install dist/*.whl
/tmp/t/bin/python -c "from orthomni_codeco import OrthoMNIModel; print(OrthoMNIModel().correct('xin chao'))"

twine upload --repository testpypi dist/*        # try on TestPyPI first
twine upload dist/*                              # real PyPI

License

MIT