orthomni-codeco

Vietnamese spelling correction on CPU with INT8 inference, no PyTorch required. The model is a Seq2Seq BARTpho/MBart (~400M parameters).

• Fast by default: runs on CTranslate2, which keeps the encoder, the decoding loop and the KV-cache in native C++ code. ONNX Runtime is available as an alternative backend.
• Lightweight: the only dependencies are ctranslate2, numpy, sentencepiece and huggingface-hub (versus ~1.3 GB with torch + transformers).
• Automatic model download: the wheel contains no weights. On first run, only the files of the selected backend (~400 MB) are downloaded from the Hugging Face Hub to ~/.cache/orthomni_codeco/, with a progress bar. Later runs work offline.
• Batteries included: FastAPI server, CLI and Dockerfile.

Installation

pip install orthomni-codeco            # library (CTranslate2 backend)
pip install "orthomni-codeco[server]"  # + FastAPI/uvicorn
pip install "orthomni-codeco[onnx]"    # + ONNX Runtime backend (optional)

Requires Python >= 3.9.

Quickstart

from orthomni_codeco import OrthoMNIModel

model = OrthoMNIModel()                         # first run: downloads the model to the cache, then loads it (<1 s)
model.correct("hom nay troi dep qua")           # 'Hôm nay trời đẹp quá.'
model.correct(["xin chao", "toi di hoc"])       # ['Xin chào.', 'Tôi đi học.']

predict() and generate() are aliases of correct(). Corrector is the old name of OrthoMNIModel.

Library usage

from orthomni_codeco import OrthoMNIModel, CorrectorConfig, InputTooLongError, get_corrector, ensure_model

model = get_corrector()                                   # per-process singleton, config read from ORTHOMNI_* env vars
model = OrthoMNIModel(CorrectorConfig(intra_op_threads=4))
model = OrthoMNIModel(CorrectorConfig(model_dir="/path/to/int8"))  # use a local model, no download

await model.acorrect("xin chao")                          # async, does not block the event loop

try:
    model.correct(very_long_text)
except InputTooLongError as e:
    print(e.indices, e.max_tokens)                        # which inputs exceed the limit -> split sentences first

ensure_model()                                            # only download the model to the cache, returns its path

• Load once: create OrthoMNIModel once and reuse it.
• Thread-safe: calls are serialized with a lock, since the backend already uses all the threads it is given.
• Lightweight import: import orthomni_codeco does not load ctranslate2, onnxruntime or numpy.

Backends

Backend	Install	Model files	Notes
ct2 (default)	included	ct2/model.bin (~400 MB)	CTranslate2 INT8; encoder, greedy decoding and KV-cache all run in C++
onnx	pip install "orthomni-codeco[onnx]"	onnx/*.onnx (~470 MB)	ONNX Runtime INT8; decoding loop in Python

backend="auto" (default) picks ct2 when ctranslate2 is installed, otherwise onnx. Choose explicitly with CorrectorConfig(backend="onnx"), ORTHOMNI_BACKEND=onnx or --backend onnx on the CLI. Both backends use the same tokenizer. The current model on the Hub (package >= 0.1.2) ships only ct2/; the onnx backend needs a local model directory exported with --backends onnx.

Model source

In order of priority:

1. model_dir / ORTHOMNI_MODEL_DIR: a local model directory, nothing is downloaded.
2. repo_id + revision (ORTHOMNI_REPO_ID, ORTHOMNI_REVISION): downloaded from the Hugging Face Hub to cache_dir (ORTHOMNI_CACHE_DIR, default ~/.cache/orthomni_codeco/<org>--<repo>/<revision>/).

Private repos use HF_TOKEN or hf auth login. For offline environments, pre-download with orthomni download, then set HF_HUB_OFFLINE=1.

Mounting into an existing FastAPI app

import asyncio
from contextlib import asynccontextmanager
from fastapi import FastAPI
from orthomni_codeco import get_corrector
from orthomni_codeco.server import create_router

@asynccontextmanager
async def lifespan(app: FastAPI):
    await asyncio.to_thread(get_corrector)   # load the model at startup
    yield

app = FastAPI(lifespan=lifespan)
app.include_router(create_router(), prefix="/spell")   # -> POST /spell/predict, GET /spell/health

CLI and standalone server

orthomni download                                   # download the model to the cache, print its path (--backend onnx for ONNX)
orthomni correct "toi di hoc" "xin chao"            # quick test
orthomni serve --port 8006                          # workers/threads derived from the vCPU count
orthomni serve --workers 1 --threads 8              # lowest latency

python -m orthomni_codeco ... is equivalent to orthomni ....

Method	Path	Description
GET	/health	200 {"status":"ok"} when the model is ready, 503 otherwise
POST	/predict	Body {"text": str} or {"texts": list[str]}

curl -X POST localhost:8006/predict -H 'content-type: application/json' -d '{"text":"hom nay troi dep qua"}'
# {"result":"Hôm nay trời đẹp quá.","latency_ms":113.7}

Error codes: 422 invalid input (both text and texts, batch too large, empty text), 413 sentence longer than MAX_INPUT_TOKENS (the server does not truncate), 500 inference error.

Configuration (environment variables)

Variable	Default	Meaning
ORTHOMNI_MODEL_DIR	(empty)	Local model directory; when set, no download happens
ORTHOMNI_REPO_ID / ORTHOMNI_REVISION	see config.py	Hugging Face Hub repo and revision
ORTHOMNI_CACHE_DIR	~/.cache/orthomni_codeco	Model cache directory
ORTHOMNI_BACKEND	auto	auto, ct2 or onnx
ORTHOMNI_INTRA_OP_THREADS	0 (= vCPU / workers)	Compute threads within one operator (both backends). Biggest effect on latency
ORTHOMNI_INTER_OP_THREADS	1	(onnx) Parallelism across graph branches
ORTHOMNI_COMPUTE_TYPE	int8	(ct2) CTranslate2 compute type
WEB_CONCURRENCY	vCPU / 4	Number of uvicorn workers; each holds its own model copy (~1.7 GB RAM)
ORTHOMNI_MAX_INPUT_TOKENS	256	Maximum tokens per sentence
ORTHOMNI_INFER_BATCH_SIZE	8	Sub-batch size (sorted by length to reduce padding)
ORTHOMNI_NEW_TOKENS_RATIO / _OFFSET / _CAP	1.3 / 8 / 320	max_new_tokens = min(CAP, len_input × RATIO + OFFSET)
ORTHOMNI_WARMUP	true	Run one dummy inference on load
ORTHOMNI_MAX_BATCH_SIZE	32	Maximum number of texts per request
ORTHOMNI_MAX_TEXT_CHARS	4000	Maximum characters per text
ORTHOMNI_MAX_CONCURRENCY	1	Concurrent generate requests per worker

Avoid thread thrashing: keep workers × ORTHOMNI_INTRA_OP_THREADS ≤ vCPUs. The serve command reads CPU limits from cgroups (docker --cpus, K8s limits). Fewer workers with more threads gives lower latency; more workers with fewer threads gives higher throughput.

Docker

Requires the INT8 model in ./model, laid out like the Hugging Face repo (ct2/, onnx/, tokenizer files; see below):

docker build -t orthomni-codeco:cpu .
docker run --rm -p 8006:8006 --cpus 8 --memory 4g orthomni-codeco:cpu

Development

Environment setup

python -m venv .venv && . .venv/bin/activate
pip install -e ".[server,dev]"
ORTHOMNI_MODEL_DIR=./model pytest -q        # model tests are skipped if ./model is missing
./run.sh                                    # local server using ./model, port 7428

Exporting a new model

pip install torch --index-url https://download.pytorch.org/whl/cpu   # CPU-only torch first
pip install -e ".[export]"
orthomni export --model_dir /path/to/checkpoint --output_dir ./model [--backends ct2 onnx] [--target avx512_vnni]

• ct2/: CTranslate2 INT8 (all linear layers and embeddings). Independent of the CPU architecture.
• onnx/: dynamic INT8 per-channel on MatMul (per-tensor quantization produced incorrect sentences), then compacted: embeddings become INT8 per-row, and the cross-attention K/V projection moves into the encoder so decoder_model.onnx (a ~90% duplicate of decoder_with_past_model.onnx) is dropped. ONNX INT8 kernels depend on the CPU, so export on a machine with the same architecture as the serving machine (--target).

To convert a 0.1.0 model directory (3 ONNX files) without the original checkpoint, use python scripts/convert_v010_model.py --src ./old_model --dst ./model. The script rebuilds an FP32 checkpoint from the INT8 weights; since they already sit on the symmetric INT8 grid, CTranslate2 re-quantizes them to the same values.

Publishing the model to the Hugging Face Hub

hf auth login
python scripts/upload_model.py --model_dir ./model --repo_id <org>/orthomni-codeco

The script prints a commit hash. Update DEFAULT_REPO_ID and DEFAULT_REVISION in src/orthomni_codeco/config.py before releasing, so each package version always downloads the same set of weights.

Build and publish

pip install -U build twine
rm -rf dist/ && python -m build                  # -> dist/*.whl + dist/*.tar.gz
twine check dist/*                               # check metadata and that the README renders on PyPI
unzip -l dist/*.whl                              # confirm no model files (.onnx/.bin) are inside

# Try the wheel in a clean venv
python -m venv /tmp/t && /tmp/t/bin/pip install dist/*.whl
/tmp/t/bin/python -c "from orthomni_codeco import OrthoMNIModel; print(OrthoMNIModel().correct('xin chao'))"

twine upload --repository testpypi dist/*        # try on TestPyPI first
twine upload dist/*                              # real PyPI

License

MIT