Pantogloss

Pantogloss is a TensorFlow/Keras many-to-English machine-translation library. Its first model, pantogloss-500-en, was converted and numerically validated from the model described in Many-to-English Machine Translation Tools, Data, and Pretrained Models (ACL-IJCNLP 2021).

The Python package is distributed through PyPI, while the initial model is kept in a separate public Hugging Face repository. Pantogloss 0.3.0 and later download it anonymously by default; cached or explicit Hugging Face credentials remain supported for private and gated model repositories.

The codebase and converted model are licensed under Apache-2.0. This repository is private during initial development.

Intended API

from pantogloss import Translator

translator = Translator.from_pretrained("pantogloss-500-en")
print(translator.translate("Comment allez-vous ?"))

Greedy decoding remains the default because the full evaluation found it about 16 times faster than beam-4. The validated quality preset enables beam-4 with length penalty 0.6 without changing the return type:

print(translator.translate("Comment allez-vous ?", preset="quality"))

Use preset="fast" for the explicit greedy preset, or preset="custom" with beam_size and length_penalty for advanced tuning. Explicit decoding values also override either standard preset.

The CLI exposes the same choice:

pantogloss translate --preset fast "Comment allez-vous ?"
pantogloss translate --preset quality "Comment allez-vous ?"
pantogloss translate --preset custom --beam-size 8 --length-penalty 1.0 \
  "Comment allez-vous ?"

Pantogloss selects the first TensorFlow GPU automatically and enables memory growth. Device choice can also be made explicit:

translator = Translator.from_pretrained("pantogloss-500-en", device="gpu")
print(translator.device_info)

Using device="gpu" fails clearly if TensorFlow cannot see a GPU; use device="cpu" to force CPU inference.

Greedy and beam translation use encode-once, graph-compiled TensorFlow decoding loops with decoder self-attention and cross-attention key/value caches by default. If a TensorFlow backend cannot compile a loop, Pantogloss falls back to its eager cached decoder. Eager execution can also be selected explicitly:

translator = Translator.from_pretrained("pantogloss-500-en", compiled_decode=False)

For exact comparison with the original full-prefix beam calculation, construct the translator with cached_beam=False. This slower numerical reference may choose different near-tied beam candidates than incremental cached attention.

Structured results are additive to the original string API:

result = translator.translate_detailed("Bonjour le monde.")
print(result.text, result.source_tokens, result.target_tokens)
print(result.execution_device, result.elapsed_seconds)

Caller-supplied language hints can attach versioned benchmark evidence and opt-in, model-independent warnings to structured results:

result = translator.translate_detailed(
    "ສະບາຍດີ",
    source_language="lo",
    include_warnings=True,
)
print(result.language_quality.tier, result.language_quality.measured_chrf)
for warning in result.warnings:
    print(warning.kind, warning.message)

Pantogloss does not detect the source language. source_language accepts an ISO 639-1 or BCP-47 hint from the caller; a batch may provide one hint or an aligned sequence. Tiers are aggregate evidence from one pinned FLORES+ corpus, not per-sentence confidence or a guarantee. Unmeasured languages are labeled unmeasured, never unsupported. measured-higher is chrF 60 or above, measured-mixed is 45 through 59.99, and measured-limited is below 45. Warnings are disabled by default and can flag measured-limited or unmeasured languages, exact source copies, retained source scripts, repetition, empty output, and extreme length ratios.

For extracted documents, one shared model can segment, batch, and reconstruct text while retaining blank lines and paragraph boundaries:

from pantogloss import DocumentTranslator

documents = DocumentTranslator(translator)
result = documents.translate(
    extracted_text,
    source_language="fr",
    max_source_tokens=512,
    long_input="split",
)
print(result.text)

Long segments can use split, truncate, or error policy. Failures are isolated to individual segments and recorded in result.segments; successful neighbors remain ordered. Layout-only segments such as page numbers, dot leaders, and separator rules are preserved verbatim instead of being sent to the translation model.

Install pantogloss[tika] to use the pantogloss-tika extraction-to-English command without a translation server or Docker container. It supports bounded trials and page ranges, displays progress, writes output incrementally, and resumes from a fingerprint-protected JSONL checkpoint. See examples/tika_to_english.py for its compatibility wrapper.

The same workflow is available as a stable Python API:

from pantogloss.tika import translate_document

result = translate_document(
    "report.pdf",
    output="report.en.txt",
    device="auto",
)
print(result.diagnostics)
print(result.runtime)

Apple Silicon GPU document runs automatically use short-lived TensorFlow workers, committing five chunks before each worker exits. This bounds Metal's retained unified-memory allocations without changing translations or checkpoint compatibility. Tune the interval with metal_worker_chunks=N in Python or --metal-worker-chunks N on the command line; use None in Python or 0 on the command line to disable recycling. CPU and CUDA runs remain in-process.

New checkpoints record per-chunk diagnostics, translation timing, throughput, peak process memory, and effective device metadata. Inspect or validate them offline without importing TensorFlow or Tika:

pantogloss-tika inspect report.en.txt.jsonl
pantogloss-tika validate report.en.txt.jsonl --output report.en.txt
pantogloss-tika review report.en.txt.jsonl --output review.jsonl
pantogloss-tika review report.en.txt.jsonl --format csv --output review.csv
pantogloss-tika review report.en.txt.jsonl \
  --include-preserved formula --format csv --output formulas.csv

Diagnostics are conservative review signals—not translation-quality scores. They flag empty output, retained multi-character source-script runs, fourfold word repetition, and extreme character-length ratios; scientific symbols such as α, β, and π are intentionally not treated as untranslated prose. The model-free review command joins every finding back to its aligned source and translation with chunk, segment, character-offset, token, truncation, and error metadata. JSONL is the default for automated processing; CSV is convenient for spreadsheet review. Use --include-preserved formula, layout_only, or all to add informational rows for deliberately untranslated spans without turning them into diagnostic warnings. During classifier development, tools/audit_formula_detection.py replays the explainable formula policy over an existing checkpoint and can emit candidate-level JSONL with signal counts, natural-word count, and math density.

Formula-heavy spans are preserved verbatim by default because general-purpose translation models can turn extracted equations into plausible but invented prose. Checkpoints record these spans with preservation_reason="formula", and inspect reports preservation counts. Normal prose containing occasional mathematical notation remains translatable. Whole-span preservation requires zero natural-language words; mixed prose and notation stays in the translation and diagnostic path rather than silently retaining source-language prose. Use preserve_formulas=False with DocumentTranslator.translate() or translate_document(), or pass --translate-formulas to pantogloss-tika, to restore the prior behavior. The Tika choice is protected by the checkpoint fingerprint.

Schema-1 checkpoints created by Pantogloss 0.5 remain inspectable and can be resumed when their source checksum, model revision, and decoding options match.

Install the accelerator backend for the machine:

# Linux with an NVIDIA GPU
pip install 'pantogloss[cuda]'

# Apple Silicon
pip install 'pantogloss[metal]'

Both use the same device="auto" or device="gpu" Python API. The CUDA extra does not install or replace the host NVIDIA driver. The Metal extra uses Apple's TensorFlow PluggableDevice and the TensorFlow 2.18 runtime combination validated by the Bytewise project.

Source batches use bounded power-of-two padded widths by default (for example, 16, 32, and 64 through the active source-token limit). Padding remains masked and does not change source token counts. This bounds accelerator allocation shapes for long document runs; use source_padding="exact" with Translator.from_pretrained() or --source-padding exact as a reference mode. The policy and actual padded width are included in runtime and segment metadata and the policy is protected by Tika checkpoint fingerprints.

Greedy decoding runs on the selected device. On Apple Silicon, beam decoding uses a correctness-first CPU execution fallback because Panto-500 validation found shape-sensitive corruption in Metal beam-expanded inference. CUDA beam decoding remains on GPU. Pantogloss records the effective beam execution device in evaluation manifests instead of silently claiming Metal placement.

The model is stored separately in the public Hugging Face repository chrismattmann/pantogloss-500-en; it is never included in the Python wheel.

The default token=None uses a locally cached Hugging Face credential when one exists but does not require one for public repositories. Use token=False to force anonymous access or pass a token explicitly without storing it:

import os

translator = Translator.from_pretrained(token=os.environ["HF_TOKEN"])

Command line

The pantogloss command loads the model once and supports arguments, files, and line-oriented Unix pipelines:

pantogloss info
pantogloss translate "Comment allez-vous ?"
printf 'Hola señor\nWie geht es Ihnen?\n' | pantogloss translate --device gpu
pantogloss translate --input source.txt --output english.txt --batch-size 16
pantogloss translate --beam-size 4 --length-penalty 0.6 "Hola señor"
pantogloss translate --max-source-tokens 512 --source-length-policy truncate "..."
pantogloss translate --input source.txt --preserve-empty-lines \
  --report timing.json --output english.txt
pantogloss translate --json --source-language fr --include-warnings \
  "Comment allez-vous ?"

Use --json for ordered JSON Lines records containing the source index, translation, error, inference time, and preservation status. Empty input lines are omitted by default; --preserve-empty-lines copies them without model inference. Batch failures are recursively isolated so successful neighbors are retained and the command exits 1 after writing all results; --fail-fast restores immediate termination. --input-encoding, --output-encoding, and --encoding-errors control file decoding and encoding. Progress is automatic on terminals and can be forced or disabled with --progress or --no-progress. When --source-language is supplied, JSONL also includes the normalized hint and pinned benchmark evidence. Add --include-warnings for conservative output review signals. These flags never add prose to plain translation output.

Use --offline to require an already cached model snapshot. Translation data goes to stdout (or --output); model and device diagnostics are suppressed by default so pipelines remain clean. Use --verbose for Pantogloss loading progress, --report for a JSON timing/throughput summary, or --tensorflow-logs for TensorFlow, CUDA, and Metal startup diagnostics.

Development status

The complete 307-variable Keras model has been converted locally from all 308 learned PyTorch tensors (the target embedding and output projection are tied). Greedy parity against the archived RTG implementation passes across a ten-language batch: token IDs and translations match exactly, while final logits have a maximum absolute error of 1.24e-5. With the original beam size 4 and length penalty 0.6, all decoded four-best candidate sets match. One near-tied example changes top rank because of framework floating-point ordering. Model version 0.1.0 is released in the public Hugging Face repository at an immutable commit.

The source model and generated artifacts stay under the ignored artifacts/ directory. To reproduce conversion after acquiring the source archive:

python tools/convert_rtg_checkpoint.py \
  artifacts/source/rtg500eng-tfm9L6L768d-bsz720k-stp200k-ens05 \
  artifacts/converted/pantogloss-500-en-candidate

Run the reference parity harness with:

CUDA_VISIBLE_DEVICES=-1 python tools/check_parity.py \
  artifacts/source/rtg500eng-tfm9L6L768d-bsz720k-stp200k-ens05 \
  artifacts/converted/pantogloss-500-en-candidate

To require and verify real GPU placement:

python tools/check_gpu.py artifacts/converted/pantogloss-500-en-candidate

Platform validation

CUDA, CPU, and Apple Metal validation commands, benchmark methodology, release results, memory measurements, and known backend caveats are maintained on the Platform Validation wiki page. The measured-language table, tier definitions, and interpretation guidance live on the Language Quality Catalog wiki page. This keeps changing hardware evidence out of the package overview while preserving one reproducible validation record. Contributors preparing a release should follow the release checklist.

Project boundary

Pantogloss translates text and ordered text segments. It does not identify languages, detect file types, or parse document formats. The existing DocumentTranslator is a neutral text boundary; pantogloss-tika remains a frozen compatibility example rather than a direction for new core dependencies. See the architecture boundary for how external Tika, Bytewise, language-identification, and future neural parsing projects should compose with Pantogloss.

Translation evaluation

Pantogloss includes a versioned evaluation runner and a checksum-pinned, project-authored CC0 smoke corpus covering 12 languages and seven scripts. It supports durable resumable translation artifacts, model-free rescoring, adaptive batch recovery, deterministic paired-bootstrap confidence intervals, and aligned regression comparisons. Reports include BLEU, chrF, per-language diagnostics, failures, empty and unknown-token outputs, latency, and throughput.

Install the development extras and run the complete automated suite:

# Linux CUDA
python -m pip install -e '.[cuda,evaluation,test]'

# Apple Silicon Metal
python -m pip install -e '.[metal,evaluation,test]'

python -m pytest

The targeted weak-language fixture adds two CC0 regression examples each for Lao, Yoruba, Hausa, Igbo, Khmer, and Burmese, plus a public-safe per-language diagnostic exporter. See evaluation/README.md for commands, checked-in reports, reproducibility details, and the important limits on interpreting this deliberately small regression fixture.

Detailed translation-quality methodology and aggregate results live in evaluation/README.md; platform-specific execution results live only on the Platform Validation wiki page.