thalovant-languages

Every word a Thalovant component's rule turns on, for every language the sources know, derived from those sources and kept in one place, out of every codebase.

A voice satellite waits on "turn off the" because "the" is a continuation word. An intent listing closes "quelle heure est-il" with a question mark because "quelle" opens a question, and reads "volume {level} percent" aloud as "volume cinquante pour cent" in French. A French synthesiser is told to say "onze minutes" where it would swallow a consonant. Each of those is a fact about a language, not about the program that needs it. Here they are data: one directory per language, 270 of them, read by the SDK and by the satellite, and none of it written by hand.

Where the words come from

Every generated file starts with a header naming its sources, their versions, the treebanks it was read from, and the thresholds that decided what got in. The thresholds are numbers a reader can argue with; the argument then has its facts. Two of them were settled by measurement on the voice satellite: English "is" ends three sentences in a hundred and is worth waiting on, "on" ends six ("turn it on") and is not.

Layout

src/thalovant_languages/languages/
  scripts.yaml            what a writing system does that no word list can
  en/language.yaml        English, every region
  fr/language.yaml        French
  pt/language.yaml        Portuguese ...
  pt-PT/language.yaml     ... and what European Portuguese does differently
overrides/
  en.yaml, fr.yaml        what only a person knows, laid on top
  scripts.yaml            the two marks the synthesiser splits on that Unicode does not list
scripts/derive.py         the generator

A directory is named by a BCP-47 tag: bare when what it says holds for every region, regional when a source says the region differs (CLDR gives pt-PT its own plural rule). A regional file carries only the difference; the loader lays it over the language's file.

Use

import thalovant_languages as languages

languages.language("fr-CA")["continuation_words"]   # the French file
languages.words("en", "trailing_words")             # lower-cased, as a set
languages.plural_category("ru", 21)                 # "one", by CLDR's rules
languages.language("tlh")                           # {} -- nothing describes it
languages.script_pattern("unspaced").search("今天") # a character of an unspaced script
languages.marks("sentence_ends", "spaced")          # every Sentence_Terminal mark, and "…;"

A language is found by the matcher the rest of OVOS uses (ovos_spec_tools.language): fr-CA reads fr, en-GB reads en, and a language nothing describes gets an empty mapping rather than another language's rules. The keys a component reads are thalovant_languages.KEYS.

THALOVANT_LANGUAGES_DIR=/path/to/checkout/src/thalovant_languages/languages reads a checkout instead of the installed data, for trying a change before it is released; languages.refresh() forgets what was read after changing it.

Regenerating

pip install -e ".[dev]"
curl -LO https://www.unicode.org/Public/UCD/latest/ucd/PropList.txt      # and Scripts.txt, SpecialCasing.txt
curl -LO https://raw.githubusercontent.com/unicode-org/cldr-json/main/cldr-json/cldr-core/supplemental/plurals.json
# the treebanks: https://universaldependencies.org/ -> the current release archive (~700 MB)
python scripts/derive.py --ud ud-treebanks-v2.18 --cldr plurals.json --unicode .
thalovant-languages check
git diff --stat

The generator reads about four gigabytes of treebanks in a minute or two and rewrites every file; review the diff, commit. To change a word, do not edit the file: change the generator's thresholds, or put what you know in overrides/, and rerun. thalovant-languages check refuses a key no component reads, a pattern that does not compile, a plural rule the evaluator cannot read, and a list holding a boolean.

Who reads it

• thalovant (the Python SDK): thalovant.listing sets a registered pattern the way a person reads it, ranks phrases, and reads slots aloud.
• thalovant-voice (the satellite): the semantic endpoint's continuation words, the lower-casing of an all-capitals transcript, the plural forms of its own counted text, the synthesiser's pronunciation repairs.

Nothing here parses dates, numbers or colours; the OVOS parsers do that for skills on the hub. Contractions come from ovos-utterance-normalizer, which already carries them.