dormouse

Ukrainian ↔ English bridge for LLM pipelines. Normalizes surzhyk and slang, translates chat-register Ukrainian into English on the way in, and back into Ukrainian on the way out. Runs offline on CPU.

UA: Міст українська ↔ англійська для LLM-пайплайнів. Нормалізує суржик і сленг, перекладає розмовну українську в англійську на вході й назад на виході. Працює офлайн, на CPU.

---

What this actually buys you

Two things, measured, in this order of importance:

1. Small local models become usable in Ukrainian. A 3B model answering Ukrainian directly produces broken orthography and invented words. The same model, driven in English through dormouse, produces clean Ukrainian — and does it in roughly half the wall-clock time.
2. Real but modest input-token savings on cloud APIs: 11-30% depending on the provider's tokenizer. Not 60-73%.

Honest correction to earlier versions of this README

Previous releases claimed 47.5% savings on Claude and a 60-73% headline. Those numbers came from scripts/tokenize_benchmark.py, which uses local proxy tokenizers — cl100k_base stood in for Claude, Gemma-2 stood in for Gemini. Both proxies were wrong in the same direction: they overstate the cost of Cyrillic on models whose real tokenizers handle it far better.

Everything below is re-measured against the providers' own billing counters via OpenRouter (usage.prompt_tokens). The old script is kept in the repo for reference; the numbers it produces are superseded.

---

Benchmark 1 — real input tokens (OpenRouter, 2026-08-15)

22 Ukrainian prompts (12 general assistant prompts from the v0.6 agent-pipeline demo set, 10 real e-commerce customer messages) sent twice to every model: once raw Ukrainian, once translated to English by dormouse-mt-uk-en v0.7. 352 requests total. Token counts are the providers' own, not a local estimate.

Reproduce: OPENROUTER_API_KEY=… python scripts/tokenize_openrouter.py Raw data: data/exports/tokenize_openrouter.jsonl / .txt

As billed (includes each provider's fixed chat-template overhead):

target model	UK in	EN in	saved
OpenAI GPT-4	728	513	29.5%
OpenAI GPT-5.5	573	483	15.7%
OpenAI GPT-4.1	595	505	15.1%
Google Gemini 3.7 Flash	431	369	14.4%
Google Gemini 3.1 Pro	431	369	14.4%
Anthropic Claude Opus 5	701	625	10.8%
Anthropic Claude Sonnet 5	701	625	10.8%
Anthropic Claude Opus 4.8	701	625	10.8%

Content only (per-model chat-template overhead measured with a one-character prompt and subtracted from both columns — this is the saving you get on the text itself, before per-message framing dilutes it):

target model	UK in	EN in	saved
OpenAI GPT-4	574	359	37.5%
OpenAI GPT-5.5	441	351	20.4%
OpenAI GPT-4.1	441	351	20.4%
Google Gemini 3.7 Flash	431	369	14.4%
Google Gemini 3.1 Pro	431	369	14.4%
Anthropic Claude Opus 5 / Sonnet 5 / Opus 4.8	569	493	13.4%

Reading this honestly

• Legacy GPT-4 / GPT-3.5 (cl100k) is the only place where translation is a real cost lever — ~30-38%. That tokenizer genuinely punishes Cyrillic.
• Modern frontier tokenizers already handle Ukrainian well. GPT-5.5, Gemini 3.x and Claude land in the 11-20% band. On a short chat message that is a handful of tokens. If your only goal is saving money on Claude or GPT-5.5, dormouse is not worth the added latency.
• These figures are for the MT path only (dormouse-mt-uk-en translating the raw text). The rule-based squeeze() layer compresses further by removing fillers and intensifiers; that combined pipeline has not yet been re-measured against real provider counters, so no number is claimed for it here.

---

Benchmark 2 — round-trip through a small local model

The result that actually justifies the project.

Setup: qwen2.5:3b on ollama, CPU only, same shop-assistant system prompt in both arms, 10 real customer messages.

• CHAIN — UA → dormouse-mt-uk-en → qwen (English system prompt) → dormouse-mt-en-uk → UA
• DIRECT — the same UA message straight to qwen with a Ukrainian system prompt and an explicit "answer in Ukrainian" instruction

Reproduce: python scripts/roundtrip_local_llm.py qwen2.5:3b Raw data: data/exports/roundtrip_qwen3b.txt

DIRECT Ukrainian breaks at the orthography level

A 3B model does not have enough Ukrainian to stay inside the language:

"Цей перстень складений з сріbullі та латуні."          ← Latin letters mid-word
"...підвищити кваліtat стосунк з клієнтами."            ← same, plus broken case
"Пожалуйста, дайте мені детальніше..."                  ← Russian leaking in
"...я розглядаю винунацію за злиття застібки."          ← invented words
"Ваше замовлення №1042 від {{data.date}}..."            ← template variable emitted
"Дзвінкайте, я перевірю..."                             ← not a word

CHAIN output is fluent

UA in    хочу замовити два браслети, знижка якась є на два?
EN in    I want to order two bracelets, is there any kind of discount for two?
EN out   We offer no specific discount but welcome your order!
CHAIN UA Ми не пропонуємо конкретної знижки, а вітаємо ваш заказ!

UA in    а можна оплатити при отриманні? бо карткою не хочу
EN in    Can I pay when I receive it? Because I don't want to pay by card.
EN out   Yes, you can pay upon receipt. We accept all major cards for payment.
CHAIN UA Так, ви можете оплатити чек. Ми приймаємо всі основні картки на оплату.

And it is faster

Two extra 76M translator passes cost less than making a 3B model generate Cyrillic token by token:

arm	per-message wall clock (10 messages, CPU)
CHAIN	22.7 - 49.3 s (median ~40 s)
DIRECT	49.6 - 95.4 s (median ~70 s)

Takeaway: the pitch is not "save money on Claude". It is "run a 3B model locally and still serve Ukrainian customers" — no API key, no data leaving the machine, GDPR-clean, and better output than the same model produces on its own.

---

Known failure modes

Read this before shipping dormouse into anything customer-facing. The MT models are trained on chat and generic customer-support text and inherit that domain. Nouns outside it drift, and a wrong noun survives the round trip intact:

input	translated as	should be
реквізити (bank details)	refunds	payment details
гравіювання імені	name-playing order	name engraving
застібка (clasp)	zip → стільниковий ремонт	clasp repair
brass	мідь (copper)	латунь
Necklaces	краватки (neckties)	намиста
ring	обручка (wedding ring)	перстень

Also unfixed: ти/ви mixing in the en→uk direction — English has no T-V distinction, so the reverse model picks a register at random within one reply.

Mitigation today: pass domain terms through a protected-span glossary before translation (the placeholder-masking machinery already exists in pii.py). A domain-specific e-commerce fine-tune is the proper fix and is not done yet.

---

MT model quality (v0.7, held-out eval, sacrebleu)

Fine-tunes of Helsinki-NLP/opus-mt-uk-en and opus-mt-en-uk, 76M params each, CPU inference. Training corpus for v0.7 is 285 784 pairs (uk→en) and 282 945 pairs (en→uk), mixed from four sources:

source	uk→en	en→uk	what it is
v06/inv	158 482	158 521	real Ukrainian chat (Telegram, Threads) — human
native	96 738	97 912	native English chat (hh-rlhf, oasst1), UA side MT
cs	20 924	24 702	customer-support bitext, UA side MT
surzhyk	9 640	1 810	synthetic surzhyk injected into real sentences

uk→en — Dariachup/dormouse-mt-uk-en, 1000 held-out pairs:

slice	n	base BLEU	ft BLEU	base chrF	ft chrF
overall	1000	23.20	38.46	45.49	58.80
cs *	250	24.82	58.99	47.94	75.70
native *	250	34.01	47.77	57.53	67.63
v06	250	17.43	31.32	38.82	53.16
surzhyk	250	16.12	25.79	37.97	47.36

en→uk — Dariachup/dormouse-mt-en-uk, 845 held-out pairs:

slice	n	base BLEU	ft BLEU	base chrF	ft chrF
overall	845	21.12	40.28	44.25	62.90
cs *	250	27.49	68.15	49.35	82.49
native *	250	26.93	49.43	49.75	68.97
inv	250	9.70	19.97	33.97	45.26
surzhyk	95	9.47	14.16	36.55	42.43

Read the starred slices with suspicion

* — on the cs and native slices the Ukrainian side is machine-generated (gemini-2.5-flash), not written by a human: it is the source in uk→en and the reference in en→uk. A 48-68 BLEU there means "the fine-tune reproduces Gemini's Ukrainian well", which is what it was trained to do. It is not evidence of human-level quality.

The honest slices are v06 / inv (real Ukrainian chat with human references) and surzhyk. Read those first: 31.32 BLEU uk→en and 19.97 BLEU en→uk.

That comparison also corrects an earlier claim in this README. The overall en→uk number (40.28) is now higher than uk→en (38.46), which looks like the reverse direction caught up — it did not. Strip the synthetic-reference slices and the picture is unchanged: 31.32 vs 19.97. Generating Ukrainian morphology is still the harder half of the round trip.

Against cloud translators — 39 real chat samples, every hypothesis scored against the same human reference (measured on v0.6; not yet re-run for v0.7):

model	BLEU	chrF	cost / 40	offline
Mistral Nemo 12B (cloud)	34.48	52.68	~$0.003	❌
Qwen3-235B (cloud)	33.79	54.40	~$0.005	❌
dormouse-mt-uk-en 76M	28.48	49.55	$0	✅
Qwen 2.5 7B (cloud)	24.74	43.55	~$0.006	❌
opus-mt-uk-en base 76M	22.52	41.36	$0	✅

A 76M offline model beats generic Qwen 2.5 7B on this domain and sits ~6 BLEU behind flagship 12B+ cloud translators, with zero API cost and no network.

---

How it works

graph LR
    A[UA text<br/>surzhyk, slang] --> B[crack_open<br/>normalize]
    B --> C[compress<br/>remove fillers]
    C --> D[map_to_en<br/>lexicon + MT]
    D --> E[EN<br/>for the LLM]

    style A fill:#fdd,stroke:#c33
    style E fill:#dfd,stroke:#3a3

Layer	What it does	How
crack_open	surzhyk, slang, profanity → standard UA	360 rules + pymorphy3 lemmatization
compress	remove fillers, intensifiers, noise	rule-based pattern matching
map_to_en	UA → English	47K lexicon + seq2seq, or MarianMT v0.7

Install

pip install dormouse-ua

Lexicon (47K entries), seq2seq model (28K expression pairs) and vocab files are bundled. The MarianMT fine-tunes are downloaded from HuggingFace on first use and pinned by sha256 in assets.py.

pip install dormouse-ua[ml]      # + torch, sentence-transformers
pip install dormouse-ua[all]     # everything

Quick start

from dormouse import squeeze

# Normalize only (layers 1+2)
squeeze("шо там по баґу, пофікси плз")
# → "що там по помилці, виправ"

# Cloud mode — normalize and map to English (layers 1+2+3)
squeeze("ваще нормально, канєшно зробимо", target="cloud")
# → "generally ok, sure do"

Direct MT access

from dormouse.mt_translator import get_translator

uk_en = get_translator("uk-en")
en_uk = get_translator("en-uk")

en = uk_en.translate("а можна оплатити при отриманні?")
# → "Can I pay when I receive it?"
en_uk.translate("Yes, you can pay upon receipt.")
# → "Так, ви можете оплатити при отриманні."

SDK middleware (drop-in)

from openai import OpenAI
from dormouse import DormouseClient

client = DormouseClient(OpenAI())  # or Anthropic()

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "шо там по деплою, він ваще не робе"}],
)
# squeeze → EN → model → unsqueeze → Ukrainian

Classification and search (offline, no API)

from dormouse import sniff, stir, mumble, sip

sniff(["Борщ український", "Чізкейк Нью-Йорк"],
      {"Гарячі страви": "борщ суп юшка", "Десерти": "торт чізкейк еклер"})

stir("report.pdf")                                    # index
mumble("холодні закуски")                             # search by meaning
sip("data.xlsx", topics=["HR", "finance"])            # classify

MiniLM-L12-v2 embeddings, CPU, no keys, no cost.

CLI

dormouse squeeze "шо там по баґу" -t cloud
dormouse stir book.pdf
dormouse mumble "головний герой"

---

Comparison with alternatives

Every general-purpose prompt-compression tool operates on already-English text. dormouse works one level earlier, on the Ukrainian side.

tool	Ukrainian	approach
dormouse	native	normalize + compress + translate
LLMLingua	no	GPT-2 perplexity pruning
Selective Context	no	self-information filtering
token-reducer	no	6-stage pipeline

On a shared 20-prompt Ukrainian set, LLMLingua removed ~10% of tokens — its GPT-2 perplexity model does not read Cyrillic well enough to prune it. That comparison was made with local tokenizers and is being re-run against real provider counters; treat the exact percentages as provisional.

Use cases

• Local, private Ukrainian assistants — the strongest case. Run a 3B model on your own hardware and still get fluent Ukrainian. Nothing leaves the box.
• Chatbots and support — users write in surzhyk and slang; normalize before the model sees it.
• RAG — user searches in slang, documents are in literary Ukrainian. Normalize both sides and match by meaning.
• Legacy GPT-4 / GPT-3.5 pipelines — the one place the token saving is large enough to matter on its own.
• Offline search and classification — stir / mumble / sip need no API.

Eval details

Corpus:         53,351 texts (Telegram + books)
Squeeze speed:  606 texts/sec (normalization)
Seq2seq model:  7.3M params, 28K expression pairs
MT models:      76M params each, MarianMT, CPU inference
Stir/mumble:    8,441 chunks indexed, search ~600 ms

Quality-preservation scores from earlier releases (99-102% across the GPT-4.1 family) were produced by a heuristic length-and-structure judge, not an LLM judge. They are directionally useful, not precise, and are not reprinted here as headline claims.

Architecture

src/dormouse/
├── optimizer.py       — squeeze() main pipeline
├── unsqueeze.py       — EN → UA on the way back
├── mt_translator.py   — MarianMT fine-tunes (uk-en, en-uk)
├── rule_engine.py     — normalization (360 rules + pymorphy3)
├── compressor.py      — filler/noise removal
├── classifier.py      — sniff() embeddings-based classification
├── mapper.py          — UA→EN via lexicon + lemma + transliteration
├── seq2seq.py         — expression translator (GRU encoder-decoder)
├── teapot.py          — stir/mumble/sip/brew (search + LLM)
├── embedder.py        — sentence-transformers wrapper
├── middleware.py      — OpenAI/Anthropic SDK proxy
├── cli.py             — Click CLI
├── assets.py          — bundled data + sha256-pinned model download
└── data/              — lexicon.db, seq2seq model, vocab, rules

Development

git clone https://github.com/ChuprinaDaria/dormouse
cd dormouse
pip install -e ".[dev,morph]"
DORMOUSE_DATA_DIR=./data pytest tests/ -v

Benchmarks:

OPENROUTER_API_KEY=… python scripts/tokenize_openrouter.py   # real token cost
python scripts/roundtrip_local_llm.py qwen2.5:3b             # local round-trip

License

MIT

---

Built by Daria Chuprina because she can 👾.

Lazysoft | LinkedIn | dchuprina@lazysoft.pl