thalovant-languages

Every word a Thalovant component's rule turns on, for every language the sources know, derived from those sources and kept in one place, out of every codebase.

A voice satellite waits on "turn off the" because "the" is a continuation word. An intent listing closes "quelle heure est-il" with a question mark because "quelle" opens a question, and reads "volume {level} percent" aloud as "volume cinquante pour cent" in French. A French synthesiser is told to say "onze minutes" where it would swallow a consonant. Each of those is a fact about a language, not about the program that needs it. Here they are data: 292 language and locale directories, read by the SDK and by the satellite, and none of it written by hand.

Where the words come from

Every generated file starts with a header naming its sources, their versions, the treebanks it was read from, and the thresholds that decided what got in. The thresholds are numbers a reader can argue with; the argument then has its facts. Two of them were settled by measurement on the voice satellite: English "is" ends three sentences in a hundred and is worth waiting on, "on" ends six ("turn it on") and is not.

Layout

src/thalovant_languages/languages/
  scripts.yaml            what a writing system does that no word list can
  en/language.yaml        English, every region
  fr/language.yaml        French
  pt/language.yaml        Portuguese ...
  pt-PT/language.yaml     ... and what European Portuguese does differently
overrides/
  en.yaml, fr.yaml        what only a person knows, laid on top
  scripts.yaml            the two marks the synthesiser splits on that Unicode does not list
scripts/derive.py         the generator

A directory is named by a BCP-47 tag: bare when what it says holds for every region, regional when a source says the region differs (CLDR gives pt-PT its own plural rule). A regional file carries only the difference; the loader lays it over the language's file.

Use

import thalovant_languages as languages

languages.language("fr-CA")["continuation_words"]   # the French file
languages.words("en", "trailing_words")             # lower-cased, as a set
languages.plural_category("ru", 21)                 # "one", by CLDR's rules
languages.asks("on mange à quelle heure ce soir", "fr")  # True: it holds a question word
languages.language("tlh")                           # {} -- nothing describes it
languages.script_pattern("unspaced").search("今天") # a character of an unspaced script
languages.marks("sentence_ends", "spaced")          # every Sentence_Terminal mark, and "…;"

A language is found by the matcher the rest of OVOS uses (ovos_spec_tools.language): fr-CA reads fr, en-GB reads en, and a language nothing describes gets an empty mapping rather than another language's rules. The keys a component reads are thalovant_languages.KEYS.

Since 0.3.1, Traditional Chinese tags (zh-TW, zh-HK, zh-MO, zh-Hant) find a zh-Hant word list. Simplified Chinese tags can use zh-Hans. Both inherit shared Chinese rules. These are writing-system variants, not Cantonese translations. asks("什麼時候出發", "zh-TW") works without inserting spaces or a question mark.

SkillKit owns skill dialog translations and regional resource generation. This package supplies shared language rules; it does not need a copy of every skill's regional folders.

THALOVANT_LANGUAGES_DIR=/path/to/checkout/src/thalovant_languages/languages reads a checkout instead of the installed data, for trying a change before it is released; languages.refresh() forgets what was read after changing it.

Regenerating

pip install -e ".[dev]"
curl -LO https://www.unicode.org/Public/UCD/latest/ucd/PropList.txt      # and Scripts.txt, SpecialCasing.txt
curl -LO https://raw.githubusercontent.com/unicode-org/cldr-json/main/cldr-json/cldr-core/supplemental/plurals.json
# the treebanks: https://universaldependencies.org/ -> the current release archive (~700 MB)
python scripts/derive.py --ud ud-treebanks-v2.18 --cldr plurals.json --unicode .
thalovant-languages check
git diff --stat

The generator reads about four gigabytes of treebanks in a minute or two and rewrites every file; review the diff, commit. To change a word, do not edit the file: change the generator's thresholds, or put what you know in overrides/, and rerun. thalovant-languages check refuses a key no component reads, a pattern that does not compile, a plural rule the evaluator cannot read, and a list holding a boolean.

The Chinese treebanks contain both writing systems. After deriving zh, the generator uses OpenCC 1.4.2 to produce deduplicated zh-Hans and zh-Hant word lists. OpenCC is only needed for development; it is not a runtime dependency. To regenerate those lists from the committed Chinese source, without downloading treebanks again:

python scripts/derive_chinese.py
python scripts/derive_chinese.py --check

The check fails when a generated list is missing or stale. Tests run it too.

Install and check a release

python -m pip install --upgrade thalovant-languages
thalovant-languages check

PyPI receives releases first. Piwheels publishes Raspberry Pi wheels independently, so a new release may appear there later. The package itself is pure Python; its wheel also works across platforms.

Who reads it

• thalovant (the Python SDK): thalovant.listing sets a registered pattern the way a person reads it, ranks phrases, and reads slots aloud.
• thalovant-voice (the satellite): the semantic endpoint's continuation words, the lower-casing of an all-capitals transcript, the plural forms of its own counted text, the synthesiser's pronunciation repairs.
• the fleet's fallback skills (weather, date and time): a keyword net claims a sentence only when asks() says it is a question.

Nothing here parses dates, numbers or colours; the OVOS parsers do that for skills on the hub. Contractions come from ovos-utterance-normalizer, which already carries them.