Kintsugi

Red-teaming for machine translation. Kintsugi finds where a translation model breaks: inputs that make it hallucinate drug names, erase a negation, leak its prompt, refuse, soften violent text, or silently switch language. It is named for the Japanese art of repairing breakage with gold, so the cracks can be seen and mended.

It packages a battery of translation attacks, a failure detector, and judges:

• 201 attack categories / 2822 seeds across 9 target languages, each tagged with the catastrophic/bad rates actually observed in the runs.
• A 123-check local detector that catches obvious failures (empty, copy through, refusal, prompt leak, repetition, script mismatch, number drop, negation erasure, plus per-language traps) with no model call.
• 11 adversarial source transforms (homoglyphs, zero-width and RTL controls, fullwidth, injection, negation dropping, repetition, mixed scripts).
• 115 translator framings (personas, task reframings, multi-hop pivots, context dilution) that change failure rates.
• A two-layer judge (local prefilter then an LLM judge) that cut judge error from 53% to 0.2% in the project.
• Frequent-vs-rare word attacks for 20 bundled languages: the same probe on common vs rare words, to measure the resource-tier effect directly.

It is provider-agnostic. Point it at a local Ollama model, any Hugging Face model, your own model via a plain function, or a frontier API. Every finding records the model's provenance (provider, model, version).

All bundled data is shipped as Python literals, not data files: importing kintsugi opens no files and parses no JSON, CSV or pickles.

Install

pip install kintsugi                 # core: zero dependencies
pip install "kintsugi[huggingface]"  # to target Hugging Face models
pip install "kintsugi[frequency]"    # frequency probes beyond the 20 bundled languages

Python 3.9+.

Quick start

Try your own model in the simplest possible way: give a function.

from kintsugi import RedTeam, CallableProvider, load_attacks

target = CallableProvider(lambda text, src, tgt: my_model.translate(text, tgt))
rt = RedTeam(target, lang_pair="en-de")                 # detector-only, no judge
findings = rt.run(load_attacks(category="prompt_injection"))
print(rt.summary(findings))

A local open-source model via Ollama, judged by a frontier model:

from kintsugi import RedTeam, OllamaProvider, AnthropicProvider, load_attacks

rt = RedTeam(
    target=OllamaProvider("qwen3.5:0.8b"),
    judge=AnthropicProvider("claude-opus-5-5"),   # set ANTHROPIC_API_KEY
    target_lang="Japanese", lang_pair="en-ja",
)
findings = rt.run(load_attacks(category="japanese_pharma_traps"), progress=True)
print(rt.summary(findings))

Any Hugging Face model, the simplest way to try an open-weights model:

from kintsugi import RedTeam, HuggingFaceProvider

rt = RedTeam(HuggingFaceProvider("Qwen/Qwen2.5-0.5B-Instruct"), lang_pair="en-de")
print(rt.summary(rt.run(load_attacks(limit_per_category=2))))

Frequent vs non-frequent words

Catastrophic failure scales inversely with how well-represented something is in training data, and the same attack is more effective on rarer material. Kintsugi makes that a controlled probe, with frequency bands bundled for 20 languages. Japanese is the reference language (the probes carry a gold expected answer from a rank-ordered frequency dictionary); Kannada and Cantonese come from the FLORES+ low-resource benchmark.

from kintsugi import RedTeam, OllamaProvider, frequency

rt = RedTeam(OllamaProvider("qwen3.5:0.8b"), lang_pair="en-ja")
bands = frequency.contrast("ja", n=50)                  # frequent vs rare, held constant
common = rt.summary(rt.run(bands["frequent"]))
rare   = rt.summary(rt.run(bands["rare"]))               # expect a higher catastrophic rate

Bundled languages: ar de en es fr hi id it ja kn ko nl pl pt ru sv tr vi yue zh zh-Hant.

Locale variants: region, script, register

Models often collapse a variant to its base language, and that collapse is a failure worth catching. kintsugi.locales registers region, script, and register (diglossia: standard vs colloquial/creole) variants as red-team targets, each resolving to a frequency band:

from kintsugi import locales, RedTeam, OllamaProvider

locales.diglossia_pairs()            # e.g. ('en-SG','en-SG-x-singlish'), ('de-CH','gsw'), ('ar','ar-EG')

std      = locales.get("en-SG")              # Singapore Standard English
singlish = locales.get("en-SG-x-singlish")   # Singlish (creole)
rt = RedTeam(OllamaProvider("qwen3.5:0.8b"),
             source_lang="English", target_lang=singlish["name"], lang_pair="en-en")

Covered: region (pt-BR/pt-PT, fr-CA/fr-CH, de-AT/de-CH, zh-CN/ zh-TW/zh-HK/zh-MO/zh-SG, ...), script (zh-Hans/zh-Hant, sr-Cyrl/ sr-Latn, ms-Latn/ms-Arab, hi-Deva/hi-Latn, ...), and register (Singapore Standard English vs Singlish, Indian English vs Hinglish, Jamaican Standard English vs Patois, Swiss Standard German vs Schweizerdeutsch, MSA vs Egyptian/Moroccan Arabic, HK written Chinese vs spoken Cantonese).

kintsugi list-locales --register creole
kintsugi list-locales --script Hant
kintsugi frequency --target ollama:qwen3.5:0.8b --lang-of zh-Hant --band rare --n 30

The full Google Translate / Translated / Apple coverage catalogues are reference material in docs/LANGUAGE_COVERAGE.md, kept out of the code on purpose (that list is never-ending).

Command line

kintsugi list                                            # all attack categories
kintsugi list-framings                                   # all translator framings
kintsugi transform zero_width "Do not take with alcohol"
kintsugi attack  --target ollama:qwen3.5:0.8b --judge anthropic:claude-opus-5-5 \
                 --lang en-ja --category japanese_pharma_traps --out findings.jsonl
kintsugi frequency --target ollama:qwen3.5:0.8b --direction en-ja --band rare --n 30

What the attacks find

A few headline numbers, drawn from the bundled observed rates:

Vulnerability tracks resource tier almost perfectly: French is the hardest to break, low-resource Indic and Cantonese targets the easiest. Models at or above ~4B parameters show near-zero catastrophic rates; models at or below ~2B should not be used for safety-critical translation.

import kintsugi
kintsugi.observed_rates()["japanese_pharma_traps"]["en-ja"]   # {'n': 79, 'catastrophic': 40, 'bad': ...}

How judging works

1. Local prefilter (kintsugi.prefilter) classifies obvious failures for free and returns None for anything ambiguous.
2. LLM judge (kintsugi.Judge) scores the rest on adequacy, fluency, faithfulness, format and safety, and never returns an error verdict: on a parse failure it defaults to ok so statistics stay clean, and always-severe failure types (hallucination, refusal, prompt leak) are promoted.

A judge is optional. With no judge, kintsugi still runs the full prefilter and profanity check, which already catch the most dangerous failure modes.

License

Apache 2.0.