tetrak-translit

Detect, transliterate and fold text in scripts that have several romanisations: Armenian, Georgian and Russian so far. A dependency-free Python library and command-line tool for the problem every catalogue of such material has: the same name exists in its own script, sometimes in more than one orthography, and in three or five Latin spellings, and no stock search engine treats them as one word.

Status: stable. The API has not changed since 1.0; the major releases since followed new editions of the standards, which moved renderings. Versions say what moved: a change to what any input renders or folds to is at least a minor release, since indexes built on the fold must be rebuilt, and a change to an existing test case is a major one. The tables and the folds are tested against the sets in tests/data/, hundreds of Armenian fold cases among them; what is not in them is not claimed, and the known gaps are written down. Measured in catalogue search across scripts over about a thousand records, romanised queries in every scheme now fold to the script's key 99 to 100 times in a hundred, up from 82 at the first release.

pip install tetrak-translit

>>> import tetrak_translit as tt
>>> tt.detect("Երեւան").orthography
'classical'
>>> tt.transliterate("Երևան", "ala_lc")
'Erevan'
>>> tt.transliterate("Երևան", "bgn_pcgn")
'Yerevan'
>>> tt.transliterate("Յակոբ", "western")
'Hagop'
>>> tt.transliterate("თბილისი", "national")
'tbilisi'
>>> tt.fold("Yerevan", script="hy") == tt.fold("Երեւան") == tt.fold("Erewan", script="hy")
True
>>> tt.fold("Boghossian", script="hy") == tt.fold("Պօղոսեան") == tt.fold("Poghosyan", script="hy")
True
>>> tt.fold("Chavchavadze", script="ka") == tt.fold("ჭავჭავაძე") == tt.fold("Ch'avch'avadze", script="ka")
True
>>> tt.detect("Русскій вѣстникъ").orthography
'pre-reform'
>>> tt.fold("Tchaikovsky", script="ru") == tt.fold("Чайковский") == tt.fold("Chaĭkovskiĭ", script="ru")
True

$ tetrak-translit detect "Ազգ եւ հայրենիք" "თბილისი"
armenian (classical)	Ազգ եւ հայրենիք
georgian	თბილისი
$ tetrak-translit transliterate --scheme iso_9985 "Չարենց"
Č'arenc'
$ tetrak-translit transliterate --scheme iso_9984 --to script "Tbilisi"
თბილისი
$ cut -f2 titles.tsv | tetrak-translit fold --script hy

Three jobs

Detection. Which script a string is in and, for Armenian, which orthography: classical (everything printed before the 1922 reform, and the diaspora still), reformed (Soviet and post-Soviet Armenia) or indeterminate (the string reads the same under both). For Russian, pre-reform (the four letters and the final hard sign the 1918 reform removed, which émigré presses kept) or reformed. Unicode character names plus a few orthographic tells, chiefly a free ւ outside the ու digraph. For catalogue search across scripts, where a title's language code often says nothing, a Latin value is also read for the marks a romanisation leaves: the aspirate in Patmutʻiwn, the ĕ of Keankʻi mĕ. Detection reports the marks it finds as evidence, never as a verdict.

Transliteration, between a script and its named Latin schemes, each a versioned, tested table written from its published standard:

A scheme key alone is enough when the text or the key settles the script; otherwise say script="ka" or use the qualified id. Reversible where the standard is one-to-one; the others are documented as lossy and the library refuses to guess rather than reverse them.

The canonical fold. One function that maps any of the above, script or Latin, onto a single index form, so that index-time and query-time normalisation agree. This is what a Solr or Elasticsearch analysis chain calls. Each script owns its rules, because what should be merged differs (Armenian folds b with p, Georgian must not), so Latin input names the script it is to be folded for. The Armenian fold generalises the script fold in Tetrak's Armenian recogniser and takes the orthography findings of Tetrak brief 014 as its first test cases.

The schemes are for display; the fold is for finding. A fold key is a deliberately coarse ASCII skeleton (poghosian, kevork, haiastan) and must never be shown to a reader. Cross-script name-matching research (LREC 2020) finds there is often no gold form to score against, and this package does not pretend there is one.

What it does not do

• Translate. Transliteration is the same words in different letters.
• Guess at lossy reversals. transliterate(..., to="script") works for ISO 9985, Hübschmann-Meillet and ISO 9984 and raises for the rest.
• Fold every popular spelling. Spellings that insert an unwritten schwa (Megerdich for Մկրտիչ), come through Russian (Khachaturian for Խաչատրյան, Tiflis for თბილისი) or are translations (George for გიორგი) do not fold to the script, nor do the German and French spellings of Russian names that use a letter for another sound (Michail, Zwetajewa, Pouchkine), and a classical Armenian աւ that reformed spelling turned into օ does not either. The test sets are the claim.
• Handle every script. Armenian, Georgian and Russian so far; Syriac is next. The engine is script-agnostic and a new script is a package of tables; see adding a script.

Documentation

The user guide and API reference are at scattercode.github.io/tetrak-translit, built from the released code at each release.

Where it comes from

Built as part of Tetrak, a local-first OCR pipeline for archival material, to support two use cases at once: a catalogue search that has to match records across scripts, and Tetrak's own transcripts, where an ALA-LC rendering beside the OCR output is both a catalogue-ready field and an accessibility one (a screen reader with no Armenian voice can at least read the romanisation). Georgian came next because Tetrak's recognisers are heading there; Syriac is planned.

Contributing and licence

See CONTRIBUTING.md for the checks, the commit style and the release automation, and SECURITY.md for reporting a vulnerability. MIT licensed — see LICENSE.