altamt

Multilingual machine translation for Kinyarwanda · English · French — one model, every direction, fast on CPU and GPU.

altamt (Advanced Lightweight Translation AI Model Transformer) is a compact modern Transformer (RMSNorm · SwiGLU · Grouped-Query Attention · RoPE · FlashAttention) that:

• translates every trained direction with a single model (rw↔en, rw↔fr, en↔fr, …),
• auto-detects the input language when you don't specify it,
• translates documents of any length (a paragraph or 20 pages) while preserving paragraphs, headings, lists and line breaks,
• pivots automatically through a bridge language for pairs it was never trained on directly,
• runs efficiently on CPU (INT8 quantization, KV-cached decoding, physical-core thread tuning) and on GPU (bf16/fp16 autocast, token-budget batching).

---

Contents

1. Installation
2. Quickstart
3. The Translator — full reference
4. Translating: methods and parameters
5. Decoding techniques explained
6. Documents and long text
7. Language handling: auto-detection and pivot routing
8. Performance tuning per device
9. CLI reference
10. Result dictionaries
11. Troubleshooting

---

1. Installation

pip install altamt

# optional extras
pip install "altamt[onnx]"        # ONNX Runtime export & inference
pip install "altamt[benchmark]"   # compare against NLLB / Opus-MT baselines

Requires Python ≥ 3.9. CPU-only machines are fully supported — a GPU is never required for inference.

2. Quickstart

from altamt import Translator

t = Translator("path/to/checkpoint")          # a trained checkpoint directory

# The target language is required; the source is auto-detected when omitted.
r = t.translate("Mwaramutse nshuti zanjye", tgt_lang="en")
print(r["translated_text"])                   # Good morning my friends
print(r["detected_src"], r["route"])          # rw   rw->en

# Pin the direction explicitly:
t.translate("How are you today?", src_lang="en", tgt_lang="rw")

# Batch — directions can be mixed inside one batch:
t.translate_batch(["Mwaramutse", "Good morning"], tgt_lang=["fr", "rw"])

# A whole document (any length — layout is preserved):
r = t.translate_document(open("rapport.txt").read(), tgt_lang="en")

# A file, in one line (writes rapport.en.txt next to the input):
t.translate_file("rapport.txt", tgt_lang="en")

# Same things from the terminal:
altamt translate "Mwaramutse nshuti zanjye" --model path/to/checkpoint --tgt-lang en
altamt translate-doc rapport.txt --model path/to/checkpoint --tgt-lang en -o rapport.en.txt

3. The Translator — full reference

Translator(
    model_path,                  # checkpoint directory (required)
    tokenizer_path=None,
    device="cpu",
    quantize=False,
    beam_size=4,
    length_penalty=1.0,
    max_new_tokens=512,
    length_ratio=1.8,
    no_repeat_ngram_size=0,
    num_threads="auto",
    batch_tokens=0,
    lid=None,
    pivot_lang="en",
    add_src_lang_tag=None,
    verbose=True,
)

Parameter	Default	What it does
model_path	—	Checkpoint directory produced by training (best/ or last/). Must contain model.pt, config.json, a SentencePiece *.model and — bundled automatically by the trainer — vocab_config.json.
tokenizer_path	None	Explicit tokenizer file; by default the first *.model inside model_path is used.
device	"cpu"	"cpu" (the optimized default path), "cuda" / "cuda:N", or "auto" (CUDA when available, else CPU). Requesting CUDA when it is unavailable raises a clear error containing the exact fix (see altamt doctor).
quantize	False	Apply INT8 dynamic quantization after loading. CPU only. Typically ~2× lower latency and ~4× smaller weight matrices with negligible quality loss.
beam_size	4	Default beam width for all calls. 1 = greedy (fastest); 4 is the standard quality/speed trade-off (typically +1–2 BLEU over greedy). Overridable per call.
length_penalty	1.0	Beam-search length normalization exponent (Wu et al. 2016). >1 favors longer outputs, <1 shorter ones. Only affects beam search.
max_new_tokens	512	Hard cap on generated tokens. The effective per-batch budget is min(max_new_tokens, src_tokens × length_ratio + 16), so short inputs stay fast and long ones are never truncated mid-sentence.
length_ratio	1.8	Output/input token ratio used for that adaptive budget. Raise it (e.g. 2.5) if a language pair systematically expands a lot in translation.
no_repeat_ngram_size	0	When > 0, any n-gram of this size that already appeared in the output is banned from being generated again. 3 is a good value to stop degenerate repetition loops on noisy or out-of-domain input. 0 = off.
num_threads	"auto"	CPU threading. "auto" pins PyTorch to the machine's physical cores and disables the inter-op pool (PyTorch's default of one thread per logical core over-subscribes SMT machines and makes short-sentence decoding slower). An int sets torch.set_num_threads exactly; None leaves process defaults untouched.
batch_tokens	0	Padded tokens per generation batch for translate_batch / documents. 0 selects the per-device default: 4 096 on CPU, 32 768 on CUDA. The budget is divided by the beam width automatically.
lid	None	A custom altamt.LanguageDetector; the built-in rw/en/fr detector is used when omitted.
pivot_lang	"en"	Bridge language for pairs that were not trained directly (see §7).
add_src_lang_tag	None	Whether to prepend the <2src> source-language tag to the encoder input. Leave it at None: the setting the checkpoint was trained with is read from vocab_config.json, and a mismatch silently degrades quality with no error. Only override for experiments.
verbose	True	Emit status notices (auto-detected language, pivot routing, document progress) to stderr — stdout stays clean for piping. False for silent library use.

Other members: t.languages (tuple of supported codes), t.trained_pairs (set of directly-trained (src, tgt) pairs), t.warmup() (one tiny generation to trigger lazy kernel initialization — call it before latency-sensitive serving).

4. Translating: methods and parameters

translate(text, src_lang=None, tgt_lang=..., beam_size=None, length_penalty=None, max_new_tokens=None)

Translate a string of any length. Short inputs run as a single pass; anything longer than the model's trained sentence length is transparently segmented, batch-translated and reassembled (you just get the result — the returned dict then also carries segments, chars_in, chars_out).

• tgt_lang — required. A multilingual model cannot guess the output language you want.
• src_lang — optional; auto-detected when omitted (a stderr notice reports the detection).
• beam_size, length_penalty, max_new_tokens — per-call overrides of the constructor defaults.

translate_batch(texts, src_lang=None, tgt_lang=..., beam_size=None, length_penalty=None, max_new_tokens=None)

Translate many sentences efficiently. Internally the batch is length-sorted and packed against a token budget, so almost no compute is spent on padding; results come back in input order.

• tgt_lang may be one code for the whole batch or a list with one code per sentence — directions mix freely inside one batch, including pivot-routed rows.
• src_lang=None auto-detects per sentence.

translate_document(text, src_lang=None, tgt_lang=..., beam_size=None, length_penalty=None, max_segment_tokens=None, unwrap="auto", progress=False)

The explicit long-form entry point (what translate delegates to for long input), with full control:

• max_segment_tokens — tokens per segment; default 80 % of the checkpoint's training max_src_len, which keeps every segment comfortably inside what the model actually saw during training.
• unwrap — how multi-line paragraphs are treated:
  • "auto" (default): rejoins hard-wrapped prose (the shape of text extracted from PDF/DOCX, with a newline at every rendered line) while leaving lists, tables and headings alone;
  • "never": every source line stays its own unit;
  • "always": every multi-line paragraph is joined.
• progress — one-line progress notices on stderr.
• Source detection for documents runs over a large sample of the text (not one line), so a stray heading can't misdetect the language.

translate_file(input_path, output_path=None, tgt_lang=..., src_lang=None, **document_kwargs)

Reads a UTF-8 text/Markdown file, translates it as a document, writes the result and returns the written path. Default output: <stem>.<tgt_lang><suffix> next to the input (rapport.txt → rapport.en.txt).

5. Decoding techniques explained

Technique	Parameter(s)	When to use
Greedy decoding	beam_size=1	Fastest; ~1–2 BLEU below beam 4. Right for interactive/low-latency use and quick experiments.
Beam search	beam_size=4 (default), higher for maximum quality	Explores several hypotheses in parallel and keeps the best-scoring finished one. Cost grows with beam width; beyond ~8 the quality gains vanish.
Length normalization	length_penalty	Beam search naturally favors short outputs (fewer log-prob terms). The Wu et al. normalization divides scores by ((5+len)/6)^penalty. 1.0 is neutral-ish; raise toward 1.2–1.4 if outputs feel clipped, lower toward 0.8 if they ramble.
Adaptive generation budget	max_new_tokens + length_ratio	The per-batch budget is min(max_new_tokens, src_tokens × length_ratio + 16). This is why long sentences don't get cut off mid-way and short ones don't waste decode steps.
Repetition ban	no_repeat_ngram_size	Bans regenerating any already-emitted n-gram of that size. 3 cures the classic "the the the…" / phrase-loop failure on noisy input at negligible cost.
Minimum length	(internal, documents)	Document segments decode with a 1-token minimum so a segment can never come back empty.
INT8 quantization	quantize=True	CPU-only. Dynamic per-batch quantization of all Linear layers; ~2× speed, ~4× smaller matmuls, negligible quality change.
KV-cached decoding	always on	Decoder self-attention states are cached per step and encoder cross-attention keys/values are computed once, so per-token cost is independent of how much has been generated.
FlashAttention + GQA	always on (PyTorch ≥ 2.5, supported hardware)	Causality is declared (is_causal=True) rather than materialized as a mask, which makes the flash kernel eligible; grouped KV heads are consumed natively (enable_gqa). Zero configuration.

6. Documents and long text

A sentence-level model handed a five-page document as one sequence is far outside its training distribution — that's why naive tools translate a paragraph well and garble a report. altamt instead:

1. Splits on blank lines, keeping the exact separators, so paragraph spacing survives byte-for-byte.
2. Unwraps hard-wrapped prose (PDF/DOCX extraction artifacts) into logical paragraphs; keeps headings, bullets, numbered items and table rows as separate units.
3. Strips leading markers (- , 1. , ## , (a) ) before translation and re-attaches them after — the model translates prose, not punctuation scaffolding. Titles and initials (M. Dupont, J. R. R. Tolkien) are not mistaken for list markers.
4. Splits sentences with an abbreviation-, initial- and decimal-aware splitter tuned for English, French and Kinyarwanda (fig. 3, 3.5, Dr. Smith don't split).
5. Packs sentences into segments up to the token budget, so short sentences keep their mutual context and nothing exceeds the trained length. A single over-long sentence splits at clause boundaries first.
6. Passes through rules, bare URLs and numeric-only rows verbatim instead of hallucinating over them.
7. Batches everything: the whole document is translated in length-sorted token-budgeted batches — a 20-page document is one efficient pass, not thousands of single-sentence calls.

With an identity translation, reassembly is byte-identical to the input.

r = t.translate_document(text, tgt_lang="en", progress=True)
r["segments"], r["chars_in"], r["chars_out"]     # bookkeeping

7. Language handling: auto-detection and pivot routing

Auto-detection. When src_lang is omitted, a dependency-free character-trigram Naive-Bayes detector (with a function-word prior; microsecond latency) picks the source among the model's languages, excluding the requested target. It ships with rw/en/fr profiles and can be retrained on your own corpora:

from altamt import LanguageDetector
det = LanguageDetector.train({"rw": ["mono.rw.txt"], "sw": ["mono.sw.txt"]})
det.save("lid.json")
t = Translator("ckpt", lid=LanguageDetector.load("lid.json"))

Pivot routing. Every checkpoint records which pairs it was actually trained on (trained_pairs in vocab_config.json). A directly-trained pair decodes in one hop. An untrained pair is bridged automatically through pivot_lang (default English) — e.g. rw→fr as rw→en→fr — and the result reports the full route, the pivot language and the intermediate pivot_text. If no route exists, a ValueError lists the trained pairs. Models trained on direct data for all pairs (the recommended setup) never pivot.

8. Performance tuning per device

CPU

t = Translator("ckpt", quantize=True, num_threads="auto", beam_size=1)
t.warmup()

• num_threads="auto" — pins to physical cores; the single most common CPU misconfiguration is letting PyTorch use every SMT thread.
• quantize=True — INT8; ~2× faster.
• beam_size=1 for latency-critical paths, 4 for quality.
• Batching helps a lot even on CPU: prefer one translate_batch / translate_document call over a Python loop of translate calls.

GPU

t = Translator("ckpt", device="cuda")          # or device="auto"

• Generation runs under bf16/fp16 autocast automatically.
• The generation batch budget defaults to 32 768 padded tokens — documents and large batches saturate the GPU without further tuning; set batch_tokens to trade memory for throughput.
• INT8 dynamic quantization is CPU-only; don't combine it with CUDA.

Serving tips

• Construct the Translator once and reuse it; construction loads weights and the tokenizer.
• Call warmup() once at startup.
• verbose=False in servers; all notices go to stderr otherwise.

9. CLI reference

Every command: altamt <command> --help shows all flags.

altamt translate — text in, translation out

altamt translate "Mwaramutse" --model CKPT --tgt-lang en
echo "Mwaramutse" | altamt translate --model CKPT --tgt-lang en

Flag	Default	Meaning
--tgt-lang	required	Target language code.
--src-lang	auto	Source language (auto-detected when omitted).
--beam-size	4	Beam width (1 = greedy).
--max-new-tokens	512	Generation cap (adaptive budget below it).
--int8	off	INT8 dynamic quantization (CPU).
--device	cpu	cpu / cuda / cuda:N / auto.
--threads	auto	auto, an integer, or default.
--json	off	Print the full result dict instead of just the text.

Long input is chunked transparently — the same command works for a sentence or a whole pasted article.

altamt translate-doc — files and multi-page documents

altamt translate-doc rapport.txt --model CKPT --tgt-lang en -o rapport.en.txt
cat report.md | altamt translate-doc --model CKPT --tgt-lang rw > report.rw.md

Adds to the flags above: --output/-o (default <stem>.<tgt><suffix>; stdout when reading stdin), --segment-tokens (default 80 % of the model's trained length), --unwrap {auto,always,never}, --no-repeat-ngram N, --quiet.

altamt benchmark — reportable numbers

altamt benchmark --model CKPT --test-file test.json --beam-size 4 \
    --device cpu --int8 --output-dir results --formats md latex

Reports per direction and per system: SacreBLEU (with its citable signature), chrF++, latency (ms/sentence), output tokens/sec, peak RAM and parameter count — as report.md and a booktabs report.tex. --baseline facebook/nllb-200-distilled-600M (repeatable) runs Hugging Face baselines through the same harness (pip install altamt[benchmark]). Run once with --device cpu and once with --device cuda for both columns.

altamt train / altamt train-tokenizer — training

altamt train-tokenizer --files data.json --output tokenizer/ --languages rw en fr
altamt train --config configs/base.yaml
torchrun --nproc_per_node=8 -m altamt.cli train --config configs/base.yaml

Every configuration value doubles as a CLI flag: --training.lr 1e-4, --model.encoder_layers 16, --data.languages rw,en,fr, or the --set section.field=value form. CLI wins over the YAML/JSON file; unset values keep the file's (or built-in) defaults; the effective config is saved into the run directory. See the GitHub README for the full training guide (token-budget batching, DDP/DeepSpeed, fine-tuning, resuming).

altamt quantize / altamt export-onnx — compression & export

altamt quantize    --model CKPT --output-dir int8_model
altamt export-onnx --model CKPT --output-dir onnx_model

The PyTorch INT8 path keeps the KV cache (recommended low-latency route); ONNX export produces portable encoder.onnx / decoder.onnx graphs.

altamt doctor — GPU/PyTorch environment diagnosis

altamt doctor          # diagnose: wheel CUDA vs driver CUDA, exact fix commands
altamt doctor --fix    # reinstall the torch wheel matching your driver

10. Result dictionaries

translate / each element of translate_batch:

Key	Type	Meaning
translated_text	str	The translation.
detected_src	str	Resolved source language.
src_autodetected	bool	Whether the source was auto-detected.
tgt	str	Target language.
route	str	"rw->en", or "rw->en->fr" when pivoted.
pivot	str | None	Bridge language, if pivot-routed.
pivot_text	str | None	Intermediate translation, if pivot-routed.
latency_ms	float	Wall time (per sentence for batches).

translate_document / long inputs to translate add: segments (int), chars_in / chars_out (int).

11. Troubleshooting

Symptom	Cause & fix
tgt_lang is required	Always pass the target language — a multilingual model can't guess it.
No translation route from X to Y	The pair isn't trained and no pivot bridges it; the error lists trained pairs. Train with direct data or add a pivot leg.
Output repeats a phrase forever	Set no_repeat_ngram_size=3.
Long text came out truncated/garbled on an old version	Upgrade — long input is now segmented automatically (translate and translate-doc).
device='cuda' requested but CUDA is not available	The error includes the exact fix; or run altamt doctor --fix.
Slow on CPU	quantize=True, num_threads="auto", beam_size=1, batch your inputs, call warmup().
Wrong auto-detected language on very short text	Pass src_lang explicitly — one or two words are genuinely ambiguous.

License

Apache 2.0 — full architecture, data-format, training and benchmarking documentation in the GitHub README.