tame-mt 0.1.0

Training-aware machine translation evaluation for MT.

TAME-MT

Training-Aware Machine Translation Evaluation for Machine Translation.

TAME-MT is a command-line tool and Python package that explains MT scores in light of the data the model was trained on. It does not replace BLEU or chrF. It answers a different question:

How much of this system's measured quality is earned on test examples that are close to the training corpus?

That question matters in low-resource MT because train and test data often come from narrow domains: Bible text, government forms, education materials, health leaflets, NGO documents, oral narratives, or a small number of web sources. High BLEU on a highly exposed test set can be a valid in-domain result, but it is weaker evidence of broad general-purpose translation ability.

TAME-MT makes that visible by reporting:

Number	Meaning
BLEU, chrF	Standard system quality scores.
TM-BLEU, TM-chrF	Score of a training-set nearest-neighbor translation-memory baseline.
delta over TM	System score minus translation-memory baseline score.
SourceExposure	How similar each test source is to the closest training source.
PairLeak@0.85	How many test source/reference pairs are close to the same training pair.
Far-BLEU	BLEU on test examples far from the training source corpus.
GenGap-BLEU	Near-bin BLEU minus far-bin BLEU.

Installation

pip install tame-mt

For local development:

pip install -e '.[dev]'

Quick Start

TAME-MT expects aligned UTF-8 text files:

train.src     source side of the training corpus
train.tgt     target side of the training corpus
test.src      source side of the test set
test.ref      reference translation for the test set
system.out    system hypothesis translations

Run the full report:

tame-mt score \
  --train-src train.src \
  --train-tgt train.tgt \
  --test-src test.src \
  --ref test.ref \
  --hyp system.out

Write machine-readable artifacts:

tame-mt score \
  --train-src train.src \
  --train-tgt train.tgt \
  --test-src test.src \
  --ref test.ref \
  --hyp system.out \
  --json-out report.json \
  --segment-out segments.jsonl \
  --tm-out tm.out

For large corpora, build the training index once and reuse it:

tame-mt index build \
  --train-src train.src \
  --train-tgt train.tgt \
  --out train.tameidx

tame-mt score \
  --index train.tameidx \
  --test-src test.src \
  --ref test.ref \
  --hyp system_a.out \
  --json-out system_a.tame.json

The .tameidx bundle stores compressed native source/target indexes plus the raw training text needed for TM outputs and optional segment reports. Treat it like the original training corpus for privacy and access control. Bundle loading rejects unexpected ZIP members, duplicate names, unsafe declared sizes, excessive compression ratios, default load-memory budget violations, and unsupported native-index schema versions before deserializing native bytes. Native bytes are then invariant-checked before the index can answer queries.

If the train/test/reference setup is fixed and only hypotheses change, cache the train-aware diagnostics once and reuse them:

tame-mt audit \
  --train-src train.src \
  --train-tgt train.tgt \
  --test-src test.src \
  --ref test.ref \
  --segment-out segments.jsonl \
  --json-out audit.json

tame-mt score-cached \
  --segment-in segments.jsonl \
  --ref test.ref \
  --hyp system_a.out \
  --num-train 125000 \
  --json-out system_a.tame.json

tame-mt score-cached-batch \
  --segment-in segments.jsonl \
  --ref test.ref \
  --system system_a=system_a.out \
  --system system_b=system_b.out \
  --num-train 125000 \
  --json-out-dir tame_reports

score-cached does not rebuild the training index. That makes repeated scoring of new hypotheses close to ordinary BLEU/chrF scoring cost after the first audit. When evaluating many systems, score-cached-batch reads and validates the cached segment diagnostics once and computes the TM baseline once for the whole batch. New segment JSONL outputs are accompanied by a segments.jsonl.meta.json sidecar, and cached CLI commands validate it when present so changed normalization, retrieval, TM-zero, count, or bin settings do not silently produce a mixed-configuration report. Cached JSON reports also preserve the backend that produced the original segment artifact under backend.artifact_backend, so downstream dashboards can tell whether cached diagnostics came from native_exact, native_fast, or another retrieval mode.

Python services and notebooks can go one step further with TameScorer.prepare_from_artifacts(): validate the cached diagnostics once, keep SacreBLEU reference caches and TM baseline scores alive, then call cached.score(...) for each later hypothesis.

Try the bundled toy example:

tame-mt score \
  --train-src examples/toy/train.src \
  --train-tgt examples/toy/train.tgt \
  --test-src examples/toy/test.src \
  --ref examples/toy/test.ref \
  --hyp examples/toy/hyp.out

The Core Idea

TAME-MT builds a simple translation-memory baseline from the training corpus:

1. For each test source sentence, find the most similar training source sentence.
2. Reuse that training sentence's paired target translation.
3. Score those reused translations with BLEU and chrF.

If this simple baseline already scores well, the test set is highly training-solvable under surface nearest-neighbor reuse. That does not mean the system is bad. It means raw corpus scores should be interpreted as high-exposure or narrow-domain performance unless far-bin results support a broader claim.

How Similarity Works

TAME-MT uses tokenizer-free character n-gram Jaccard similarity by default. This keeps the method deterministic and usable for low-resource languages without pretrained models, language-specific tokenizers, or downloads.

First, text is normalized for exposure calculations:

text = unicodedata.normalize("NFKC", text)
text = text.strip()
text = re.sub(r"\s+", " ", text)

By default TAME-MT does not lowercase, strip diacritics, strip punctuation, or normalize digits.

For a normalized string (s), define (G(s)) as the set of character n-grams for orders 3, 4, and 5:

G(s) = G_3(s) \cup G_4(s) \cup G_5(s)

The similarity between two strings (a) and (b) is Jaccard similarity:

s(a,b) =
\frac{|G(a) \cap G(b)|}{|G(a) \cup G(b)|}

If both strings are empty, similarity is defined as 1.0. If only one is empty, similarity is 0.0.

Exposure Metrics

Let (u_j) be the normalized source side of training pair (j), and let (v_j) be its normalized target side. For each test source (x_i), TAME-MT finds the closest training source:

E_i^{src} =
\max_j s(x_i, u_j)

The nearest-neighbor index is:

n_i^{src} =
\min \{j : s(x_i, u_j) = E_i^{src}\}

Ties are broken by choosing the lowest training index.

For a reference translation (r_i), target exposure is:

E_i^{tgt} =
\max_j s(r_i, v_j)

Pair exposure asks a stricter question: is there one training pair whose source and target sides are both close to the test source/reference pair?

For test pair ((x_i, r_i)) and training pair ((u_j, v_j)):

p(i,j) =
\min(
  s(x_i, u_j),
  s(r_i, v_j)
)

Then:

E_i^{pair} =
\max_j p(i,j)

PairLeak@0.85 is the fraction of test examples where (E_i^{pair} \ge 0.85).

In v0.1, pair exposure reranks the union of source and target top-k candidates instead of scoring every training pair. Exact pair overlap is still computed exactly.

Translation-Memory Baseline

The translation-memory hypothesis for test source (x_i) is selected using only the source side:

j^\*(i) = n_i^{src}

h_i^{TM} =
v_{j^\*(i)}

If no candidate shares character n-grams with the test source, the default policy is:

h_i^{TM} = ""

The baseline scores are ordinary corpus metrics:

B_{TM} =
BLEU(h^{TM}, r)

\Delta B_{TM} =
B_{sys} - B_{TM}

The same definitions apply to chrF.

Important: TAME-MT never uses the reference translation to choose the TM hypothesis. References are only used for scoring and diagnostics.

Distance-Stratified Quality

TAME-MT bins test examples by source exposure:

Bin	Definition
source_exact	normalized test source appears exactly in normalized train.src
near	not exact and SourceExposure >= 0.70
medium	0.30 <= SourceExposure < 0.70
far	SourceExposure < 0.30

Each bin reports count, percentage, mean source exposure, system BLEU/chrF, TM-BLEU/TM-chrF, and delta over TM.

Generalization gap is:

g_{BLEU} =
B_{near} - B_{far}

A large positive gap means the system performs much better on train-similar examples than on train-distant examples. A small gap is not automatically good: a weak system can perform poorly in every bin.

Worked Example

Suppose a training corpus contains:

train.src: god created the heaven and the earth
train.tgt: dios creó el cielo y la tierra

And the test set contains the exact same source/reference pair:

test.src:  god created the heaven and the earth
test.ref:  dios creó el cielo y la tierra

TAME-MT will report:

SourceExposure = 1.0
source_exact = true
PairExposure = 1.0
pair_exact = true
tm_hyp = "dios creó el cielo y la tierra"

That segment contributes to source_exact, not near. If many test examples look like this, raw BLEU may be measuring training-set reuse as much as system generalization.

Now suppose another test source is similar but not exact:

test.src: and the earth was without form and void

If its closest training source is:

train.src: and the earth was without form

the example may fall in the near bin. The TM baseline will reuse the paired training translation. If that baseline gets high BLEU, the test item is partly solvable by nearest-neighbor reuse.

Reading Common Patterns

Pattern	Plain-language interpretation
High BLEU, high TM-BLEU, high PairLeak, low Far-BLEU	Raw score may overstate broad generalization. Treat as high-exposure or narrow-domain performance.
Moderate BLEU, low TM-BLEU, high delta over TM, good Far-BLEU	Stronger evidence that the system is doing more than nearest-neighbor reuse.
Low BLEU, low TM-BLEU, low Far-BLEU	The test set is not TM-solvable, and the system also performs poorly.
High BLEU, low TM-BLEU, high Far-BLEU, low PairLeak	Stronger evidence of real generalization under this test distribution.
High exposure and no far-bin data	Valid in-domain result, weak evidence for broad out-of-domain generalization.

CLI Reference

Full scoring:

tame-mt score \
  --train-src train.src \
  --train-tgt train.tgt \
  --test-src test.src \
  --ref test.ref \
  --hyp system.out

Audit a benchmark before evaluating systems:

tame-mt audit \
  --train-src train.src \
  --train-tgt train.tgt \
  --test-src test.src \
  --ref test.ref

Write translation-memory hypotheses:

tame-mt tm-baseline \
  --train-src train.src \
  --train-tgt train.tgt \
  --test-src test.src \
  --out tm.out \
  --metadata-out tm_metadata.jsonl

Score a system from cached segment diagnostics:

tame-mt score-cached \
  --segment-in segments.jsonl \
  --ref test.ref \
  --hyp system.out \
  --num-train 125000

Build a reusable training index and use it without passing train files again:

tame-mt index build \
  --train-src train.src \
  --train-tgt train.tgt \
  --out train.tameidx

tame-mt score \
  --index train.tameidx \
  --test-src test.src \
  --ref test.ref \
  --hyp system.out

Common options:

--metrics bleu chrf
--ngram-orders 3,4,5
--far-threshold 0.30
--near-threshold 0.70
--leak-thresholds 0.70,0.85,0.95
--pair-k 50
--index-mode auto
--auto-exact-cutoff 5000
--candidate-gram-limit 8
--posting-limit 500
--max-candidates 3000
--rerank-limit 1000
--min-bin-size-warning 30
--tm-zero-policy empty
--lowercase
--strip-diacritics
--normalize-punctuation
--bleu-tokenize 13a
--bleu-lowercase
--chrf-word-order 2
--verbose

Numeric options must be finite decimal values. Exposure thresholds are fractions in the closed interval ([0, 1]), so values such as 1.2, nan, inf, and empty comma-list items such as 0.70,,0.85 are rejected.

Long-running commands support --verbose, which writes stage timings to stderr without changing human reports or JSON outputs.

Segment reports do not include raw text by default. Use these only when it is safe to write the data:

--include-source-text
--include-reference-text
--include-hyp-text
--include-neighbor-text

--include-neighbor-text may write raw training text.

Plain UTF-8 text files and gzip-compressed files ending in .gz are supported for corpus inputs and text/JSONL/JSON outputs. This lets large benchmarks keep train.src.gz, segments.jsonl.gz, or system.tame.json.gz compressed without changing the command shape.

Python API

from tame_mt import ScoreConfig, TameScorer

scorer = TameScorer(ScoreConfig())

report = scorer.score_files(
    train_src="train.src",
    train_tgt="train.tgt",
    test_src="test.src",
    refs=["test.ref"],
    hyp="system.out",
)

print(report.system_scores["bleu"])
print(report.tm_scores["bleu"])
print(report.delta_scores["bleu"])
print(report.signature)

In-memory corpora:

report = scorer.score_corpus(
    train_src=train_src_lines,
    train_tgt=train_tgt_lines,
    test_src=test_src_lines,
    refs=[ref_lines],
    hyp=hyp_lines,
)

Need segment diagnostics too:

result = scorer.evaluate_corpus(
    train_src=train_src_lines,
    train_tgt=train_tgt_lines,
    test_src=test_src_lines,
    refs=[ref_lines],
    hyp=hyp_lines,
)

report = result.report
segments = result.exposures
tm_output = result.tm_hyp

Reuse a saved training index from Python:

from tame_mt import load_index_bundle, save_index_bundle

save_index_bundle("train.tameidx", train_src_lines, train_tgt_lines, scorer.config)
bundle = load_index_bundle("train.tameidx", scorer.config)
result = scorer.evaluate_index_bundle(bundle, test_src_lines, [ref_lines], hyp_lines)

JSON Output

--json-out writes a stable top-level structure:

{
  "schema_version": "0.1",
  "tame_version": "0.1.0",
  "signature": "tame-mt|v:0.1.0|...",
  "data": {
    "num_train": 125000,
    "num_test": 1000,
    "num_refs": 1
  },
  "backend": {
    "name": "native_fast",
    "native": true,
    "exact": false,
    "requested_mode": "auto",
    "resolved_mode": "native_fast",
    "index_reused": false
  },
  "quality": {
    "system": {"bleu": 31.4, "chrf": 54.2},
    "tm": {"bleu": 23.8, "chrf": 47.1},
    "delta_tm": {"bleu": 7.6, "chrf": 7.1}
  },
  "exposure": {
    "source": {"mean": 0.71, "exact_overlap": 0.042},
    "target": {"mean": 0.684, "exact_overlap": 0.038},
    "pair": {"exact_overlap": 0.031}
  },
  "bins": [],
  "generalization_gap": {"bleu": 20.7, "chrf": 19.3},
  "warnings": []
}

JSON uses fractions such as 0.184; the human report renders percentages such as 18.40%.

Reproducibility

Every report includes a deterministic signature, for example:

tame-mt|v:0.1.0|norm:nfkc_ws_case|sim:char_jaccard_3-5_set|idx:auto|backend:native_fast|tm:src_nn_top1_zero_empty|bins:far0.30_near0.70_leak0.70,0.85,0.95|pair_k:50|fast:8,500,3000,1000|metrics:bleu,chrf|sacrebleu:bleu_tok_13a_lc_0_chrf_wo_2|deps:sacrebleu_2.6.0

The signature records the TAME-MT version, normalization, similarity function, requested index mode, resolved backend, TM zero policy, bin thresholds, pair reranking top-k, selected metrics, SacreBLEU settings, and dependency versions that can affect metric scores. JSON reports also include these dependency versions under config.dependencies.

Performance Modes

Nearest-neighbor search over a training corpus is the expensive part of TAME-MT. BLEU only compares hypotheses to references; TAME-MT also needs to ask, for every test source, "what training source is most similar?"

TAME-MT has native Rust retrieval backends plus pure-Python fallbacks:

Mode	Behavior	Use
native_exact	Rust exact character n-gram Jaccard retrieval over shared-gram candidates.	Small and medium corpora when exact nearest-neighbor exposure is required.
native_fast	Rust rare-gram candidate generation plus exact Jaccard reranking of a bounded shortlist.	Large corpora and interactive audits.
python_exact	Pure-Python exact fallback.	Debugging and source installs without the native extension.
python_fast	Pure-Python bounded fast fallback.	Large-corpus fallback when native wheels are unavailable.
auto	Uses native exact/fast when the extension is installed; otherwise uses Python exact/fast.	Default.

Fast mode is approximate for nearest-neighbor retrieval. Exact normalized source, target, and pair overlap checks remain exact, and shortlisted candidates are reranked with the exact Jaccard formula. For paper-critical numbers, report the signature and consider rerunning exact mode on smaller filtered subsets or high-risk bins.

Check the installed backend with:

tame-mt doctor

On the local development machine, the OPUS-100 de-en public-corpus audit at 50,000 training pairs and 2,000 test pairs completed in about 6 seconds with native_fast; score-cached on the saved segment diagnostics took under 2 seconds for an additional hypothesis. On the 100,000 train / 2,000 test OPUS-100 de-en slice, a fresh native_fast audit took about 10.0 seconds, the one-time .tameidx build took about 9.6 seconds, and a later audit from the saved index took about 2.3 seconds with identical exposure outputs.

On a synthetic benchmark on the same machine, a 100,000 train / 2,000 test native_fast audit takes about 2.5 seconds fresh, about 0.9 seconds from a saved .tameidx, and about 0.4 seconds for cached scoring of another hypothesis end to end. A prepared cached scorer takes about 0.26 seconds to build, then about 0.16 seconds per later hypothesis; five prepared cached systems score at about 0.17 seconds per system. The compressed 100,000-pair source+target .tameidx is about 67 MB on that benchmark. At 100,000 train / 10,000 test, the same path takes about 4.3 seconds fresh, about 2.8 seconds from a saved .tameidx, about 2.2 seconds for one end-to-end cached score, and about 0.8 seconds for each later prepared cached score. The cached stage is the closest analogue to ordinary BLEU/chrF scoring because it no longer touches the training corpus. Ordered cached artifacts take a fast validation path, and whole-corpus SacreBLEU statistics are reused without copying before bin aggregation. Fresh and indexed audits use slotted per-segment objects to reduce memory pressure, native maps use randomized hashing to reduce collision-DoS exposure, evaluation retrieval runs in configurable chunks, and exposure summaries collect each side once, sort once, and use binary search for threshold counts instead of rescanning the corpus for every threshold. Source-only audits and tm-baseline queries also use top-1 source retrieval unless pair exposure is being computed.

For production evaluation, use a staged workflow:

1. Run tame-mt index build --out train.tameidx once for a fixed training corpus.
2. Run tame-mt audit --index train.tameidx --segment-out segments.jsonl once for a fixed train/test/reference setup.
3. Run tame-mt score-cached-batch for all system outputs from that setup.

The index stage avoids rebuilding source/target postings. The audit stage is train-aware and does nearest-neighbor retrieval. The cached-score stage reuses exposure and TM hypotheses, so adding another system output does not require another pass over the training corpus. Batch cached scoring also avoids re-reading segment diagnostics and recomputing TM metric baselines for every system.

Privacy

TAME-MT runs locally. It does not download models, call remote services, or send text anywhere.

Training data can still be sensitive. By default, TAME-MT does not print nearest-neighbor training text. Segment reports contain indices and scores unless raw text fields are explicitly requested.

Index bundles created by tame-mt index build store raw training source/target lines and normalized exact-match and pair keys so later runs can produce identical TM outputs and optional neighbor-text diagnostics. Do not publish .tameidx files unless the underlying training corpus can also be published. Treat .tameidx files from unknown sources as untrusted inputs. The loader enforces size, compression-ratio, and default load-memory limits, validates the archive shape, and validates native index invariants before queries can run, but the safest workflow is still to rebuild bundles from trusted train.src/train.tgt files. Use --max-index-load-bytes only for trusted large bundles on machines with enough memory.

Limitations

TAME-MT does not prove that a neural model memorized a sentence. It does not measure semantic adequacy. It does not make a high-exposure benchmark invalid. It does not replace human evaluation.

The default similarity is surface-based. It can miss semantic paraphrases and can overemphasize orthographic similarity. Pair exposure in v0.1 uses top-k candidate reranking for speed; exact pair overlap is exact.

Fast retrieval mode is approximate. It is designed to make large-corpus audits usable while retaining exact scoring within the selected candidate set.

Development

pip install -e '.[dev]'
pytest
ruff check .
mypy src/tame_mt
cargo fmt --check
cargo clippy -- -D warnings
cargo test
python benchmarks/validate_fast_recall.py --require-native
python -m build
python -m twine check dist/*

For release work, run scripts/acceptance.sh locally and see docs/release.md. Tagged releases are built by GitHub Actions, checked with twine, audited, attested, accompanied by an SBOM artifact, and published through PyPI trusted publishing only after a manual tagged workflow dispatch with publish=true. CI also runs a larger staged benchmark outside the Python-version matrix; the local acceptance script keeps the heavier 100k train / 2k test performance gate for release candidates.

Citation

@software{tame_mt_2026,
  title = {TAME-MT: Training-Aware Machine Translation Evaluation for Machine Translation},
  year = {2026},
  version = {0.1.0},
  url = {https://github.com/hunterschep/TAME-MT}
}

License

MIT.