Dilemma

Dilemma is a diachronic Greek lemmatizer spanning Ancient Greek (Classical, Homeric, Hellenistic), Medieval/Byzantine Greek (both vernacular and literary), and Modern Greek (Demotic and Katharevousa).

It is part of the Open Greek project and the NLP companion to the Open Greek Corpus, whose corrected open-licensed texts feed its corpus frequencies and form-attestation data.

A four-layer pipeline, not a single model

Dilemma stacks complementary techniques and falls back through them in priority order, so the expensive layer only runs on the words the cheap layers can't reach. This is the single most important thing to know about the design.

#	Layer	Resolves	Speed
1	14.5M-form SQLite lookup	known forms (Wiktionary + LSJ + Sophocles + GLAUx + treebanks)	O(1) hash, microseconds
2	Rule-based fallback	augment / reduplication stripping, elision, crasis, enclitic particles, compound decomposition	O(1) per rule
3	Dialect normalization	Ionic, Doric, Aeolic, Koine -> Attic, so the (Attic-heavy) lookup can match	O(k) candidates
4	Character-level transformer	unseen forms the other three layers miss	beam search, O(b·n²)

The lookup carries the bulk; layers 2-3 catch systematic things a lookup table can't enumerate exhaustively; layer 4 (a 4M-parameter encoder-decoder trained from scratch on 3.5M supervised pairs) only sees the ~5% the cheap layers can't resolve. That is why Dilemma is both faster than pipelines that route every word through a model and more accurate than ones that lean only on a lookup table.

See Pipeline overview for the full layer list with every sub-rule (particle stripping, BK-tree spell correction, etc.).

Three distinctive choices within the pipeline

A 14.5M-form lookup driven by Wiktionary's own Lua modules. The biggest contributor to layer 1's coverage is running Module:grc-decl and Module:grc-conj -- the modules Wiktionary editors invoke via templates when writing inflection tables -- via wikitextprocessor over the LSJ headword set (32K nouns + 22K verbs + 14K adjectives) and the Sophocles Byzantine/Patristic lexicon (13.5K nouns + 4.6K verbs). The current expansion plus the pinned, known-good historical expansion layer generate millions of additional Ancient Greek entries, bringing the current AG input to 10.64M mappings and the combined lookup to 14.47M forms across all languages. This mechanism closes the classical and Byzantine vocabulary gap that every other tool has. See LSJ/Sophocles expansion.

A character-level encoder-decoder for layer 4, not a fine-tuned subword LM. The SIGMORPHON architecture (4M params, ~50 MB ONNX) trained from scratch on 3.5M (form, lemma) pairs from Wiktionary inflection tables (including the LSJ verb-paradigm Lua expansion) plus the GLAUx morphologically-tagged corpus. (The other treebanks and the LSJ/Sophocles noun expansion feed the lookup table, not the model.) Existing Greek tools either fine-tune giant subword models (stanza, spaCy, ~300M+ params) or use hand-coded rule engines (Morpheus, CLTK) that don't generalize. Training from scratch takes ~35 min on a 4090; the small specialized model matches ByT5-small (300M params, 20-hour fine-tune) because Greek lemmatization is pattern-rich enough that 4M parameters of Greek-specific capacity beat 300M of general capacity. See Transformer model.

Convention remapping baked into the API. The same word has different citation forms in different dictionaries -- LSJ writes εἶπον as λέγω, Cunliffe writes γίνεται as γίγνομαι, Triantafyllidis writes σπήλαια as σπήλαιο. Other tools lock you into whatever convention their training treebank used. Dilemma(convention=...) remaps the output to LSJ, Cunliffe, Triantafyllidis, or Wiktionary at runtime from the same model weights:

Dilemma(convention="lsj")               # LSJ headwords
Dilemma(convention="cunliffe")          # Cunliffe Homeric Lexicon
Dilemma(convention="triantafyllidis")   # Triantafyllidis MG dictionary
Dilemma()                               # default: Wiktionary headwords

This is the difference between scoring 65% on a benchmark because of citation mismatch and 95% on the underlying linguistic task. See Conventions.

Worth flagging

• No other lemmatizer covers Ancient, Byzantine, and Modern Greek in one system. Morpheus is classical AG only; stanza is AG or MG but not both. Dilemma resolves Katharevousa, vernacular medieval, and regional MG varieties (Cypriot, Cretan) alongside Homer and Herodotus.
• Dialect normalization is systematic, not a wordlist. No other Greek lemmatizer handles Ionic/Doric/Aeolic/Koine variation this way.
• Elision recovery does consonant de-assimilation. καθ’ becomes κατά by reversing the aspiration rule, with corpus frequency as the tiebreaker between candidates.
• Fully supervised, end to end. Every training pair is labelled (Wiktionary tables, Lua expansions, gold treebanks) and the lookup is a dictionary of correct answers -- nothing relies on unsupervised pattern discovery from raw text. Failure modes are predictable and fixes reproducible: edit the Wiktionary entry, rerun the pipeline, the fix flows through.

SQLite + ONNX backends: The lookup loads from a pre-built SQLite database (~0.3s startup) instead of parsing 600 MB of JSON (~11s), falling back to JSON if the DB isn't present. ONNX Runtime (~50 MB) and PyTorch (~2 GB) produce identical inference results -- use whichever you already have. The lookup handles 95%+ of words and needs neither.

For unseen forms, ONNX decoding batches active beams from different words in groups of at most 128. Each word retains its own beam ranking and stopping condition. Encoder batches are also capped at 128 words to bound memory on large input lists. This reduces repeated decoder calls without changing model weights or search width. tests/test_batched_decoder.py compares the ranked results with the previous serial search, including a real-model check under --run-slow. An instance also retains up to 65,536 context-independent model guesses so lemma selection and candidate scoring do not decode the same form again. Different beam widths use separate cache entries.

Table of Contents

• Quick Start
  • Installation
  • Basic Usage
  • Conventions
  • Citation-Form Hygiene
• Evaluation
  • Multi-period benchmarks
  • Elided forms (AGDT Iliad)
  • Held-out gold: the Gorman treebanks
  • Rare vocabulary coverage
  • DiGreC treebank
  • HNC Modern Greek
• How It Works
  • Pipeline overview
  • Lookup table
  • Rule-based fallback layer
  • Dialect normalization
  • Orthographic normalizer
  • Transformer model
• API Reference
  • Language and convention options
  • Verbose mode
  • Batch processing
  • POS-aware disambiguation
  • Spelling correction
  • Paradigm generation
  • Elision expansion
• POS tagger and dependency parser
• Greek Coverage
  • Language codes
  • Modern Greek varieties
  • Ancient Greek varieties
  • Medieval/Byzantine Greek
• Development
  • Full installation
  • Training
  • LSJ/Sophocles expansion
  • Export to ONNX
  • Testing
  • Frequency-ranked inflection lists
  • Hunspell spell-check export
  • Polytonic next-word prediction LM
• Data
  • Sources and scale
  • HuggingFace Hub dataset
  • Confidence tiers
  • Quality controls
  • Related work
  • Known issues
• Architecture
• Projects using Dilemma
• Credits
• License

---

Quick Start

Installation

pip install "dilemma-nlp[onnx] @ git+https://github.com/open-greek/dilemma.git"
python -m dilemma download

The first line installs the package plus ONNX Runtime (~50 MB, for unseen-form inference). If you already have PyTorch installed, use dilemma-nlp[torch] instead, or just plain dilemma-nlp to skip the model backend and rely on the lookup table alone. The second line downloads the lookup tables, ONNX model files, and tagger weights from HuggingFace into ~/.cache/dilemma/ (~4.3 GB: lemma and model data ~2.8 GB + tagger weights ~1.5 GB; add --no-tagger for the ~2.8 GB lemma-only download). The two opt-in form-attestation DBs add ~1.36 GB more.

Dilemma uses $DILEMMA_DATA_DIR if set; otherwise it picks whichever of ~/.cache/dilemma/data/ (the download), <repo-root>/data/ (a clone), or a bundled copy holds the most recently built or downloaded lookup.db, so a fresh local rebuild is never shadowed by a stale download cache. Set DILEMMA_DATA_DIR to force a specific copy.

To work from a git checkout (for development or to rebuild the data):

git clone https://github.com/open-greek/dilemma.git && cd dilemma
pip install -e ".[onnx]"
python -m dilemma download          # or use the build pipeline below

To build the data from scratch instead of downloading:

python build_data.py --download        # downloads Wiktionary dumps, builds lookup tables
python build_lookup_db.py              # builds SQLite DB for instant startup (optional)
python fix_selfmaps.py                 # fixes inflected forms that self-map (optional)

Basic Usage

from dilemma import Dilemma

d = Dilemma()                                  # all periods (default, AG-priority)
d.lemmatize("ἀνθρώπους")                      # "ἄνθρωπος"
d.lemmatize("ἔγραψα")                         # "γράφω"
d.lemmatize_batch(["λέγουσι", "θεούς"])       # ["λέγω", "θεός"]

# Elision expansion (AG elided forms resolved via Wiktionary lookup)
d.lemmatize("ἀλλ̓")                            # "ἀλλά"
d.lemmatize("ἔφατ̓")                           # "φημί"
d.lemmatize("δ̓")                              # "δέ"
d.lemmatize("κατ̓")                            # "κατά"

# Single period
d_mg = Dilemma(lang="el")                     # MG only
d_mg.lemmatize("τραγουδάει")                  # "τραγουδάω"
d_mg.lemmatize("έγραψε")                      # "γράφω"
d_grc = Dilemma(lang="grc")                   # AG only

# Specific model scale
d = Dilemma(scale="test")                     # use test-scale model

# Treebank evaluation mode: resolve articles and personal/interrogative pronouns
d_eval = Dilemma(resolve_articles=True)
d_eval.lemmatize("τῆς")                       # "ὁ" (not "τῆς")
d_eval.lemmatize("μοι")                       # "ἐγώ" (not "μοι")
d_eval.lemmatize("αὐτοῦ")                     # "αὐτός"
d_eval.lemmatize("τι")                        # "τις" (indefinite)
d_eval.lemmatize("τί")                        # "τίς" (interrogative)

# Byzantine text with orthographic normalization
d_byz = Dilemma(normalize=True, period="byzantine")
d_byz.lemmatize("εκκλησια")                   # "ἐκκλησία" (restores accents and breathing)

By default, articles and pronoun clitics self-map (e.g. τῆς returns τῆς). This is better for alignment pipelines where you want surface-form matching. Set resolve_articles=True to resolve them to canonical lemmas (ὁ, ἐγώ, σύ, αὐτός, τις, τίς), matching treebank conventions (AGDT, DiGreC). This mode uses accent placement to distinguish indefinite τις from interrogative τίς; a final grave is normalized to acute. Default lookup can retain independent adverb and particle readings of forms such as αὐτοῦ and τί. The opt-in mode also recognizes monotonic projections of polytonic articles, such as τών from τῶν, without removing accents from unrelated words. The triantafyllidis convention auto-enables article resolution (articles to ο, skipping AG pronoun resolution for forms like σε/με that are MG prepositions).

Conventions

Different dictionaries cite the same Greek word differently. LSJ uses λέγω for the εἶπον family; Cunliffe uses γίγνομαι rather than γίνομαι; Triantafyllidis uses είμαι rather than εἰμί. A tool that outputs εἰμί against a gold standard expecting είμαι looks wrong, even though both answers are correct for their respective conventions.

The convention parameter remaps Dilemma's output to whichever standard you're targeting. The same model can serve an LSJ workflow, a Cunliffe workflow, and a Triantafyllidis workflow without retraining.

Convention	Target	Example mappings
None (default)	Wiktionary headwords	εἶπον→εἶπον, θεούς→θεός, σπήλαια→σπήλαιον
lsj	LSJ dictionary	εἶπον→λέγω, αἰνῶς→αἰνός, σπηλαίου→σπήλαιον
cunliffe	Cunliffe Homeric Lexicon	γίνεται→γίγνομαι, θέλει→ἐθέλω, νοῦν→νόος
triantafyllidis	Triantafyllidis MG dictionary	ὁ→ο, εἰμί→είμαι, σπήλαιον→σπήλαιο, εἷς→ένας

The mapping is built automatically from data/lemma_equivalences.json cross-referenced against the convention's headword list, with explicit overrides in data/convention_{name}.json. The Triantafyllidis inventory uses only the source dictionary's bare citation headwords and unions them with the existing Modern Greek Wiktionary inventory for words outside the dictionary; definitions, examples, pronunciation, grammar, etymology, and source-page content are neither read nor shipped. Lemma equivalences also group valid alternative lemmatizations (comparative/positive adjective forms, active/deponent pairs, spelling variants) so that benchmarks score them as correct rather than penalizing convention disagreements.

Note: data/lemma_equivalences.json covers Ancient Greek equivalences. For Modern Greek, the lookup merge in build_data.py prefers EL (Greek) Wiktionary lemma forms over EN (English) Wiktionary, since EL uses the modern contracted forms that reflect actual usage (τρώω, λέω) while EN tends toward fuller morphological stems (τρώγω, λέγω). For the remaining cases where the two sources use genuinely different lemmas, downstream consumers like Lemma use a separate MG equivalences file generated by cross-referencing mg_lookup_scored.json against Wiktionary headwords, with corpus frequency as a tiebreaker for the canonical form.

Cross-source form references preserve cycles in every Greek language dataset. A noun and verb can each have an inflection spelled like the other's headword: ἔργον (work) and ἔργω (enclose) retain separate lemmas and paradigms instead of collapsing according to the order of the source entries. Ancient training pairs retain each source morphological reading. Their table owners and same-part-of-speech references resolve to independently defined heads before the flat lookup: ἀλέομαι (shun) keeps its own table, while the same spelling in the conjugation of ἀλέω (grind) still belongs to ἀλέω. An explicit reference belonging only to another part of speech cannot consume an independently defined head: the noun reading of κρίνω points to κρίνον (lily), while the verb κρίνω (judge) keeps its own conjugation. References already applied within one source receive the same check. This also preserves the verb δύω (plunge) beside the Epic numeral variant of δύο (two). Dialect, contracted-form and voice-only descriptions alone do not establish a separate lexical head, including when the source labels the entry as a verb. Their inflection tables resolve to the supported parent without treating the variant description as an independent definition. A separate lexical meaning in the same entry still supports its headword. Nominal case descriptions also recognize slash-separated cases, gender and contraction labels. Verb-form descriptions with explicit grammatical qualifiers likewise remain grammatical evidence, not independent lexical definitions. Explicit spelling references, sacred abbreviations, and unexpanded inflection templates do not supply independent definitions. A form header still permits a genuine lexical meaning, such as a lexicalized plural. Reference targets can follow a colon or an explicit word label. The entire description must be a reference or a recognized grammatical note; an appended definition remains evidence of an independent meaning. English form senses may nest a translation beneath their grammatical description. Such translations do not define another head: ἦλθον retains its aorist readings under ἔρχομαι. Greek present-tense head descriptions followed by their own lexical meanings remain independent. Alternative-spelling references from the original word to a supported head of the same part of speech still canonicalize normally. When a source assigns the alternative to a different numbered etymology from an independently defined homograph, only that alternative entry follows the reference. For example, the binding and needing senses of δέω retain their own tables; the separate alternative spelling of δήω still points to δήω. A spelling-reference chain also follows explicit references before a competing inflection-table mapping, provided it reaches one positively defined head of the same part of speech. Thus the pitch spelling πίτα follows πίττα to πίσσα, while the independent pie reading keeps πίτα. Ambiguous or unsupported chains retain the previous fallback. Scalar lookup also follows an exact form page's unambiguous, same-part-of-speech reference chain when an intermediate word belongs to an unrelated lexical head. Thus παρόντος reaches the verb πάρειμι through its participle παρών, while the separate noun παρών and its genitive παρῶνος retain their own reading. Independent source words and their normalized keys are protected; conflicting references and cycles retain the previous lookup. Name references count as conflicting evidence without redirecting names, so the name forms Ἀθηναίην, Ἀθηναίης, and Ἀθηναίῃ retain their Athena reading. This priority applies to the explicit target recorded in a source reading, including structured spelling and inflection references; scalar part-of-speech mappings keep their existing choice between homographs. An Ancient comparative or superlative description alone does not establish an independent adjective head, even when it includes a translation after a colon. A separately supplied lexical meaning still establishes that head. Synthesized first-aorist active infinitives and present or future neuter participles retain their accent position while taking a circumflex on a known long penult before a short final syllable (τμῆξαι, παιδεῦον). Unmarked vowel length is not guessed, and finite optatives keep their distinct accent. A diphthong consumes only two vowels; a following vowel starts another syllable when determining the accent's permitted position. An inflected reading does not replace an independently defined homograph: δίω keeps its own conjugation as well as its separate reading in the conjugation of δείδω. data/grc_reading_heads.json records the exact source definitions, parts of speech, and source hashes for surviving Ancient training heads that differ from their flat lookup target or are absent from the flat map. The training-data integrity check requires this evidence; an unrelated definition under another part of speech cannot authorize an exception.

In the Modern Greek dumps a capitalized page (a name, an abbreviation, a heading in sentence case) reaches its lowercase keys only where no lowercase page or table claims them: the female name Ερωτώ does not take ερωτώ from the verb, nor the abbreviation ΔΕΣ take δες from βλέπω. And a form-of reference of one Wiktionary that would point a lemma back at a word the other files under it is not followed: EN files ρωτώ as a form of ρωτάω while EL's ρωτώ table lists ρωτάω, and the cycle made the chain-breaking delete both conjugations. The same priority applies across the two dictionaries: a weak lowercase key derived from a name in one source does not override the other's verb form (λεν, τρων, μιλάνε). A form page's head line and its verb inflection table resolve through the word that page names. They do not inherit a noun homograph's lemma: ερωτήσεις remains a form of ερωτώ on verb rows, and the participle forms of αντιπαθών belong to αντιπαθώ. The same POS-specific references are followed through spelling pages, so ἀντιπαθών also reaches the verb. A row tagged as a verb is omitted when its target is attested only as another part of speech; this removes misclassified noun and adjective forms without losing their genuine nominal rows. Existing noun, adjective and ethnonym paradigms remain. Only a sense with an explicit form_of reference can supply verb evidence from its gloss; descriptive mentions of a verb do not redirect passive heads or create new verb lemmas. When the gloss names that same reference, it can correct an incorrect head-page POS (δηώ). A broken reference does not remove a separately attested verb head or proper-name head form. A participle's top-level reference can resolve its verb only when that target reaches an independent lexical verb entry in either dictionary, including through an explicit spelling-reference chain. All explicit verb targets from both dictionaries are retained separately from inflection-table claims. A successful sense-level reference chain from the participle's own entries takes priority over top-level pointers. A finite-form homograph's references do not replace that parent, and a sense labeled as a synonym is not a spelling alias. Other alias nodes use their explicit verb references; table evidence is used only when neither supplies a finite endpoint. Conflicting successful targets remain unresolved. Participles tagged on an entry or one of its senses remain intermediate forms, and an explicit-reference cycle must have one unambiguous exit to a finite verb. A standalone participle that would otherwise remain a verb lemma also enters this check, even without a top-level pointer. Its actual references are followed before an unsupported verb reading is omitted. This check runs after both sources have been read, so entry order does not affect it. A resolved participle keeps that finite verb through POS-specific cleanup instead of inheriting a flat homograph's lemma; independent noun and adjective readings remain. Its own recorded table also keeps that supported verb when a spelling-alias pointer names an unresolved participle (καθούμενος to κάθομαι, despite καθήμενος). Without a supported own-head path, the unresolved alias still blocks the verb reading. Nominal labels in top-level references cannot supply verb endpoints through unrelated inflection tables. An unresolved participle also cannot remove an independently defined finite verb through a reverse table or lookup alias. The build also writes data/el_participles.json for explicitly recorded participles whose finite parent remains unresolved. This is morphology for dictionary export, separate from the verb training pairs: it preserves the source headword and its own nominal inflection cells without inventing a finite verb or relabeling it as an adjective. A headword without a table retains only its own spelling, with no inferred case or descendants. The version-1 JSON envelope has language: "el" and a cells array. Each row records form, lemma, original pos: "verb", kind (headword or inflection), grammatical tags including participle, the unmodified source_form and source_lemma, and sorted source editions in sources. Inflection rows require an explicit case and number and exclude finite person or mood cells. Different source spellings and cells remain separate; only identical readings merge their source editions. An independently defined noun's own table also retains its source lemma in the training pairs, even when the flat lookup chooses an inflectional homograph. The masculine noun σκιάς and the feminine genitive σκιάς of σκιά therefore keep separate paradigms. Form-of and alternative-spelling entries still follow their explicit references; no ownership marker enters the output JSON, and flat lookup arbitration is unchanged. A Modern noun form page with an explicit case and number does not supply a full table when that table assigns incompatible cells to the page's own headword. Its exact form reference remains: noun κρατών stays the genitive plural of κράτος, while the independently defined adjective κρατών owns κρατούν and κρατούσα. Spelling-variant pages, tables that reproduce the stated cell, and entries with a separate lexical meaning retain their tables. data/el_noun_heads.json documents the independent noun heads for which that flat choice differs, with exact source headwords, noun senses, tags, and SHA-256 hashes of the source editions. The version-1 envelope has language: "el", source_sha256, and heads; each head retains all distinct qualifying source entries in evidence. Integrity checks permit a non-self-mapped noun lemma only when this companion contains valid independent evidence for that exact head. Inflection, alternative-spelling, and misspelling entries cannot authorize an exception. Both the command-line data check and pytest validate the built pairs against the lookup and this companion. Independently defined Modern adjectives retain their own source tables through the same flat homographs. Adjective κρατών therefore owns κρατούν, κρατούσα, and κρατούσες, while noun κρατών remains the genitive plural of κράτος. A new typed adjective field preserves the previously published non-adjective fields and their scalar fallbacks; an adjective correction does not expose unrelated internal noun or adverb table mappings. data/el_adjective_heads.json records exact adjective definitions, declared head types, source forms, and source-edition hashes for non-flat owners. Certification also requires an explicit adjective head template, forms in at least two genders, or an explicitly indeclinable header. A raw adjective POS label alone cannot certify a duplicated noun or adverb entry. Definitions with insufficient morphology retain their previous processing. Grammatical form descriptions, spelling references, and empty headers cannot authorize an exception. The extractor publishes the supported ADJ target in mg_pos_lookup.json whenever it differs from the scalar fallback, including forms with only one typed reading. The scalar lookup remains unchanged. The companion is generated by the normal Modern build, included in the base data download and CI registry, and required by the built-pair integrity check. The corresponding finite-verb exceptions are recorded separately in data/el_verb_heads.json. Generate it after extraction with python build/build_el_verb_heads.py --kaikki PATH_TO_DUMPS (or pass --data PATH_TO_OUTPUT). The builder selects only non-self-mapped verb lemmas from the final pairs, reads the same source editions, and requires an independently defined finite verb for every selected head. Participle, form-only, and nominal entries cannot supply that evidence. A header or no-gloss relation alone also cannot make a terminal finite verb: its explicit source references must reach a positively defined finite parent. Missing support fails the build. These companions validate morphology ownership; they do not change the runtime lookup or its POS table. Modern reference rows retain their own sense's grammatical and usage tags, rather than reducing every reference to form-of or alt-of. Distinct source cells remain separate. Independently defined determiner and pronoun heads also retain their own POS-specific morphology when the flat homograph is a verb: πάση keeps its dative tags and determiner owner πας, while the verb πας still resolves to πηγαίνω. The extraction-generated data/el_closed_class_heads.json records exact same-POS source definitions and input hashes for non-flat heads. Built-data integrity checks require this positive evidence and an exact matching DET or PRON self-target in mg_pos_lookup.json; form-only, spelling-reference, and no-gloss entries cannot authorize an exception. The scalar lookup remains unchanged. Own determiner and pronoun sense references follow same-POS chains to a positively defined head before competing table claims: εσένα and εσένανε resolve to εσύ, without treating weak form σε as a lexical head. Different accented references keep their own roots; a normalized key with multiple supported roots retains its previous mapping. lookup.db takes these changes up when build_lookup_db.py next rebuilds it. A pronoun table that spans several grammatical persons is the shared personal-pronoun grid, whose forms are no forms of the page it stands on, and is skipped whatever its template's name: in the 2026-09 dumps EN's τα page carries it under a generic template and EL's εγώ and εσείς under a renamed one, which made εγώ, εσύ, αυτός and μου forms of τα.

A Modern Greek page that only gives another spelling of a word (the Greek Wiktionary's γιός, "πολυτονική γραφή του γιος", or μιλῶ, whose link to μιλώ kaikki does not carry; the English Wiktionary's "polytonic spelling of" pages) is read as that word's alternative form, not as a lemma: as a lemma it took the word's monotonic and accentless keys, made γιος a form of γιός in a cycle that deleted its paradigm, and filed the forms of μιλώ and αγχωτικός under μιλῶ and ἀγχωτικός. Such a page is one every sense of which names the page's own word without the marks monotonic writing drops, and for a monosyllable (one run of vowels, γιος, γεια) without its accent, by an alt_of or form_of reference or by that gloss (γιος for γιός, μουστακάτος for μουστακᾶτος): 76 pages of the Greek dump, 50 of them known only by the gloss, and 47 of the English one (build_data.spelling_target). A page that names a word of other letters (ἐτρώθη, a form of τιτρώσκω), a form_of reference of a sense tagged with a case (σοφίᾳ, "dative singular of σοφία"), and a page with a sense of its own are read as before, and so are the Ancient and Medieval Greek dumps, where a page whose monotonic spelling is another word's is an inflected form of it (the dative φωνῇ of φωνή). In each Modern Greek dump's own pass two pages that name each other as forms (αγχωτικός and αγχώδης, each "συνώνυμο του" the other, which kaikki reads as alt_of), or a page naming a word whose key already maps back to it, both stay lemmas, where following the reference made a cycle that the chain-breaking deleted with both paradigms. The merged lookup checks all proposed references before applying any of them. Reciprocal synonyms such as νησοπυριτικός and ορθοπυριτικός therefore keep both paradigms regardless of their order in the dumps. The weak pronouns μόνος shows on its headword line (μόνος μου, μας, σας) are not read as its forms, which had made σας a form of μόνος.

An abbreviation is no form of the word it abbreviates, and in the Modern Greek dumps its reference does not move the key of a lowercase word with a sense of its own (build_data.Abbreviations). The English Wiktionary gives δις and δ as abbreviations of δεσποινίς; once δεσποινίς kept its paradigm, following them made the adverb δις ("twice", and the "billion" of δισ., 20,825 monotonic tokens) and the letter δ its forms. They stay themselves, and so do α, β, ν and κ, which the same pages of letters had made forms of άντε, βόρειος, νότος and κυρία. The other abbreviations the page of one abbreviation lists as alternatives are forms of the word it abbreviates, not of the page's headword: δ's δις is a form of δεσποινίς, where filed under δ, the letter, it was a noun of lemma δ. A key no lowercase word claims still follows its abbreviation: κα goes to κυρία, whose Greek Wiktionary page is an abbreviation too, μαρ to Μάρτιος beside the surname Μαρ, and the contraction στο ("σε + το"), which the English Wiktionary also tags as an abbreviation, to σε.

Modern Greek pair deduplication retains the part of speech and complete grammatical tags of each table cell. Identical rows are removed, but the genitive, accusative and vocative singular rows of καθηγητή remain separate even though their monotonic spelling is identical. The exporter can then distinguish καθηγητῆ from καθηγητή. The JSON fields remain form, lemma, pos and tags; internal reference metadata is not written to the data files.

For individual form-to-lemma corrections where the lookup table returns the wrong lemma due to ambiguity (e.g., proper nouns beating common verbs), build_lookup_db.py has a _LOOKUP_OVERRIDES dict that hard-corrects specific entries in the database.

Other tools (stanza, spaCy, CLTK) have fixed output conventions matching their training treebanks and cannot be remapped.

# LSJ lemma convention: remap output to LSJ dictionary headwords
d_lsj = Dilemma(convention="lsj")
d_lsj.lemmatize("αἰνῶς")                     # "αἰνός" (adverb -> adjective)
d_lsj.lemmatize("εἶπον")                      # "λέγω" (aorist -> present stem)

# Cunliffe convention: remap to Cunliffe Homeric Lexicon headwords
d_cun = Dilemma(convention="cunliffe")
d_cun.lemmatize("γίνεται")                    # "γίγνομαι" (Homeric form)
d_cun.lemmatize("θέλει")                      # "ἐθέλω" (Homeric form)
d_cun.lemmatize("νοῦν")                       # "νόος" (uncontracted Homeric form)

# Triantafyllidis convention: remap to Modern Greek monotonic forms
d_mg = Dilemma(convention="triantafyllidis")
d_mg.lemmatize("σπήλαια")                     # "σπήλαιο" (not σπήλαιον)
d_mg.lemmatize("Είναι")                       # "είμαι" (not εἰμί)
d_mg.lemmatize("εργαλεία")                    # "εργαλείο" (not ἐργαλεῖον)
d_mg.lemmatize("τα")                          # "ο" (not ὁ)

Citation-Form Hygiene

Ancient Greek grave accents are positional in running text, not dictionary citation accents. By default, Dilemma rejects grave-accented citation lemmas unless replacing the grave with an acute produces an independently attested Ancient/Byzantine headword. It also rejects citation candidates with overline abbreviation marks, stranded combining marks, final elision/keraia marks, or structurally impossible accent placement (a tonal mark on a non-vowel or two tonal marks on one vowel). Multiple accents across a phrase or enclitic-bearing form are not rejected merely for being multiple. These are treated as bad lemma evidence, not spelling variants to normalize blindly.

For downstream dictionary/headword audits, use citation_policy="strict_ag". That policy keeps the default artifact checks and additionally requires AG outputs to be backed by the independent headword inventory:

d = Dilemma(lang="grc", citation_policy="strict_ag")

d.citation_status("οὐρανοστεγὴς", lang="grc")
# {"normalized": "οὐρανοστεγής", "ok": True, "reason": "grave_to_trusted_headword", ...}

d.citation_status("Παντὸς", lang="grc")
# {"normalized": None, "ok": False, "reason": "untrusted_grave", ...}

d.citation_status("α̅", lang="grc")
# {"normalized": None, "ok": False, "reason": "overline", ...}

strict_ag can reduce coverage because an unverified candidate becomes None rather than a guessed headword. It is intended for audit pipelines and dictionary lookup surfaces where no lemma is safer than a false citation form. python scripts/audit_citation_hygiene.py audits the built database. Each lookup rebuild also writes data/citation_hygiene_rejections.tsv, whose entries retain the source input, table, form, lemma, and rejection reason for every citation value removed by the build-time gate.

One rejection cannot be made per pair. A corpus tokenizer that splits a printed word at its elision mark and gives the second half the whole word's lemma leaves an ending that looks exactly like an elided word - νοντ᾽ under βαρύνω, which is the end of βαρύνοντ᾽. What gives it away is the rest of the lemma's paradigm, so build_lookup_db.py runs one whole-table pass (dilemma/elision_tails.py) over the merged and per-language tables once every source has contributed. A form is rejected only when all four hold: it ends in an elision mark, it is the ending of a longer form of the same lemma, no corpus in form_profile.db wrote it, and it is cut from that longer form by two or more characters. The last two conditions are what keep real words: an unaugmented form such as γράψατ᾽ is its augmented sibling minus one character, and an unreduplicated form such as δοῦσ᾽ is its reduplicated sibling minus two but is written by the corpus. Rejections go into the same citation_hygiene_rejections.tsv, carrying the form each tail was cut from as the evidence, so the build is reversible from the report.

That pass needs data/form_profile.db, which is an opt-in download (python -m dilemma download --with-attestation) and is not a step of the rebuild order above. Rebuild without it and the filter does nothing: the word-tails are written to lookup.db as though they were words, and the same code over the same corpora produces a different artifact depending on what is on disk. The build prints a banner when that happens, and the per-table "Dropped N corpus word-tails" line is absent.

The audit reports lookup-source-aware grc and el sections because both languages share the database's deduplicated lemma table. Ancient Greek strict headword validation is applied only to AG-sourced lemmas; valid Modern Greek multiword names are not misclassified as untrusted AG headwords. The broad multi-accent diagnostic is also split into single-token and multiword/punctuated classes, but neither class is rejected without an independent structural or headword-backed reason.

In the benchmark table, the first two Dilemma rows use the Wiktionary convention. The convention="triantafyllidis" row auto-enables article resolution (articles to ο, the αυτός demonstrative to αυτός) and outputs monotonic MG lemma forms. This is the recommended setting for Modern Greek text.

Evaluation

Multi-period benchmarks

The tables below retain the earlier full comparison across tools and conventions. The 1.3.7 artifact recheck with bench_fast.py scores 99.7% on Classical Greek and 95.0% on Katharevousa with all/ wiktionary, and 96.0% on Demotic with el/triantafyllidis.

Equiv-adjusted accuracy across four periods of Greek. All tools evaluated with the same normalization (case-folded, accent-stripped) and lemma equivalence groups (see data/benchmarks/bench_all.py).

Test sets:

• AG Classical: Sextus Empiricus, Pyrrhoniae Hypotyposes 1.1-1.8 (357 tokens, First1KGreek, CC BY-SA 4.0). Not in any UD treebank or Gorman.
• Byzantine: Swaelens et al. (2024) DBBE gold standard (8,342 tokens of unedited Byzantine epigrams, CC BY 4.0). Not in the training data of any tool in the table below.
• Katharevousa: Konstantinos Sathas, Neoelliniki Filologia (1868), biography of Bessarion (318 tokens, el.wikisource.org, public domain). No Katharevousa treebank exists.
• Demotic MG: Greek Wikipedia articles "Σπήλαιο Πετραλώνων" and "Ελαιόλαδο" (400 tokens, CC BY-SA 4.0). Not in any MG treebank. A separate dev set (251 tokens from "Μέλισσα" and "Σαμοθράκη") is also included at data/benchmarks/demotic_dev.txt.

Tool	AG Classical	Byzantine (literary)	Katharevousa	Demotic MG
spaCy el	--	31.7%	44.6%	79.9%
stanza el	--	37.4%	48.4%	87.0%
Swaelens et al. (2024)	--	65.8%	--	--
CLTK	81.2%	66.6%	74.8%	--
Morpheus (oracle)	--	71.1%	--	--
stanza grc	92.2%	71.3%	85.2%	--
Swaelens et al. (2025)	--	~74-75%	--	--
Dilemma (best convention per period)	99.7%	89.2%‡	94.7%	94.8%†

_{†For Demotic MG, lang="el" with triantafyllidis (94.8%) matches lang="all" with triantafyllidis. For MG-only workloads, lang="el" with triantafyllidis is recommended since it avoids AG lemmas (e.g. σπήλαιον) being returned for MG words that have an AG lookalike. ‡Byzantine is 89.2% from the static lookup alone (no POS); with per-token POS (the shipped grc tagger, or gold POS) it reaches 91.8%, since much of the residual is genuine syncretism (θεῶ = dat. θεός vs θεάομαι) that only context resolves.}

Cells marked -- indicate the tool doesn't support that period or wasn't tested. Morpheus "oracle" picks the best candidate from all its analyses, representing the ceiling for rule-based morphology. (The TLG lemmatizer is compared separately below, since it reports a different metric and no per-era breakdown.)

Dilemma detail by convention:

Lang	Convention	POS	AG Classical	Byzantine (literary)	Katharevousa	Demotic MG
all	wiktionary (default)	--	99.7%	89.2%	94.7%	79.5%*
all	wiktionary (default)	gold	--	91.8%	--	--
all	triantafyllidis	--	87.7%	79.3%	89.6%	94.8%†
grc	wiktionary (default)	--	99.7%	88.7%	94.7%	79.5%*
grc	triantafyllidis	--	91.0%	83.3%	91.8%	89.2%
el	wiktionary (default)	--	95.2%	84.5%	91.5%	90.0%
el	triantafyllidis	--	87.7%	79.3%	89.6%	94.8%

_{*Demotic MG scores with wiktionary convention are convention mismatches, not real accuracy gaps: AG citation forms like σπήλαιον don't match the MG gold standard σπήλαιο. Using convention="triantafyllidis" fixes this. The AG and Byzantine benchmarks run with resolve_articles=True (the correct setting for gold that lemmatizes articles, e.g. τὸν -> ὁ); the article paradigm is otherwise excluded from the lookup so MG function words don't leak to AG forms.}

lang="all" searches both AG and MG lookup tables for every token. Other tools in the comparison table are locked to a single language. The wiktionary convention outputs polytonic AG citation forms. The triantafyllidis convention outputs monotonic MG lemma forms and is the recommended setting for Modern Greek text (see Conventions).

POS column: -- means Dilemma disambiguates on its own (default). gold means gold-standard POS tags from the dataset are fed in. Only DBBE provides gold POS; the +2.6pp gain there (89.2% -> 91.8%) is genuine syncretism that only context can resolve (θεῶ = dat. θεός vs θεάομαι), which the shipped grc tagger recovers without gold tags.

The eval scripts (eval/eval_dbbe.py, eval/eval_digrec.py, eval/eval_hnc.py, eval/bench_dbbe.py, eval/eval_iliad_elision.py, eval/eval_gorman_gold.py, eval/eval_ptnk.py) provide per-POS breakdowns and error categorization. eval_ptnk.py measures Koine/LXX gold agreement against the manually lemmatized UD_Ancient_Greek-PTNK Septuagint treebank (96.8% with PTNK's lemma conventions neutralized locally; its text, though not its annotation, overlaps dilemma's corpora).

Elided forms (AGDT Iliad)

Homeric Greek elides constantly, and elided forms are exactly where a frequency-ranked fallback picks the wrong homograph (elided ὅτ' is ὅτε, never the more frequent ὅτι, which does not elide in Homer). Measured over every elision-marked token in the AGDT Iliad treebank (10,312 tokens, eval/eval_iliad_elision.py, point ILIAD_TREEBANK at your AGDT tlg0012.tlg001 file):

	agreement with AGDT gold
raw	99.0%
counting elided ὅτ'/ὅθ' -> ὅτε as correct	99.3%

The adjustment is applied because AGDT itself labels its 35 elided ὅτ'/ὅθ' tokens as ὅτι, which Homeric phonology rules out. The result is independent of which apostrophe codepoint the text uses (U+2019, U+1FBD, U+02BC, ASCII, or combining U+0313). The residual is genuinely contextual: a handful of locative ὅθ' (= ὅθι) and τόθ' (= τόθι) readings, and AGDT tokenization quirks (οὐ + δ' with gold οὐδέ on the δ' token).

Held-out gold: the Gorman treebanks

The Gorman treebanks (Vanessa Gorman, CC BY-SA 4.0; 554K hand-annotated tokens across 18 classical prose authors) are deliberately never ingested into any Dilemma artifact - not the lookup, not the POS tables. They serve as the project's held-out gold standard, so agreement with their lemma annotation is a genuinely independent accuracy measure (eval/eval_gorman_gold.py, equiv-adjusted, same scoring as the multi-period benchmarks). The holdout runs deep: GLAUx sentences whose manual trees were sourced from Gorman are skipped at build time too (build/nc_filter.py), and holding all of it out costs nothing measurable - every other benchmark is unchanged. The benchmark is diachronically broad: oratory, history, philosophy, and the only held-out Ionic gold.

Author (tokens)	Gold agreement
all 18 authors (554,440)	94.0%
Herodotus, Ionic (29,327)	92.2%
Thucydides (28,923)	93.6%
Xenophon, Cyropaedia (36,116)	92.4%
Polybius (95,531)	94.6%
Diodorus (23,159)	96.2%

The residual is dominated by lemmatization-convention differences (κεῖνος vs ἐκεῖνος, plural ethnonym lemmas, voice conventions), the same class the equivalence table cannot fully absorb.

Dilemma vs the TLG lemmatizer

The Thesaurus Linguae Graecae (TLG) runs the only other diachronic Greek lemmatizer of comparable scope. It is proprietary, bound to the subscription TLG corpus and search engine, and cannot be run on external text. The only figure it publishes is a recognition rate (the share of wordforms its lexicon + paradigm analyzer can identify): 98.4%, including Byzantine and early-modern forms (Pantelia 2022). It does not report lemma-disambiguation accuracy.

To compare on equal footing, Dilemma's recognition rate is measured the same way: the share of wordforms it resolves via lookup + deterministic rules, with the neural model excluded (TLG has none, and the model maps almost any string to a nearest headword). Measured over the full held-out First1KGreek corpus (24.4M word tokens, ancient + patristic, not a Dilemma lookup source):

	TLG lemmatizer	Dilemma
Recognition rate (wordforms analyzed)	98.4%¹	98.8%²
Lemma accuracy (correct lemma chosen)	not published	94.6%³
Era coverage	ancient → Byzantine (to ~1669)	ancient → Byzantine → Modern (incl. Demotic)
Open source	No	Yes
Runs on arbitrary text	No (TLG corpus only)	Yes

_{¹ Recognition over the TLG's own subscription corpus (Pantelia, Thesaurus Linguae Graecae: A Bibliographic Guide to the Canon of Greek Authors and Works, UC Press, 2022). ² Dilemma lexicon + rules only (no neural model, to match TLG), token-level over 24.4M held-out First1KGreek tokens (Dilemma normalizes lunate sigma ϲ→σ/ς internally, since these editions use it throughout). The remaining 1.2% unrecognized is almost entirely editorial sigla, Greek numerals, and abbreviations (γρ, ΙΙ, κζ), not vocabulary gaps. Enabling the neural model lifts the full pipeline to ~99.7%. ³ Unweighted mean of the four per-period accuracy scores above (token-weighted, 90.0% - the DBBE epigrams dominate the token count) - a metric TLG does not publish.}

The two systems are on par on recognition. Beyond that, Dilemma additionally reports lemma accuracy, extends into Standard Modern Greek (Demotic), is open source, and runs as a library on arbitrary text rather than only inside a subscription corpus.

Rare vocabulary coverage

Following SIGMORPHON shared task methodology for out-of-vocabulary evaluation, we take every distinct non-capitalized word of a text, stratify by corpus frequency, and check whether the output lemma is a valid LSJ/Wiktionary headword (bench_rare.py). This tests the hard tail that matters for real texts.

Text	Period	Morpheus	Stanza	Dilemma
Xenophon, Cyropaedia	Attic	99.5%	84%	99.6%
Kresadlo, Astronautilia 13	Epic	74%	74%	84%
Herodotus, Histories	Ionic	99.5%	88%	99.3%

_{Herodotus is measured over the full Histories text (Perseus canonical-greekLit TEI, tlg0016.tlg001, CC BY-SA; 24,862 distinct non-capitalized forms). The figure holds across the frequency range: 99.6% for high-frequency forms down to 99.0% for forms attested 1-9 times, and 64/64 corpus-unseen forms. The Cyropaedia and Astronautilia rows and the Morpheus/Stanza columns are pinned to their original 2026-03 measurement. For gold-match accuracy against hand-annotated treebanks (including Cyropaedia at 92.4% and Herodotus at 92.2%, both genuinely held out), see the Gorman held-out gold benchmark.}

DiGreC treebank

On the DiGreC treebank (Homer through 15th century Byzantine Greek), Dilemma reaches 93.3% equiv-adjusted (89.3% strict) over 87,672 tokens. The gap accounts for convention differences between annotation schemes (e.g. εἶπον/λέγω, ἐγώ/ἡμεῖς). DiGreC's Herodotus, New Testament, Septuagint, and Sphrantzes sections derive from the PROIEL treebank (CC BY-NC-SA), which this project avoids entirely, evaluation included, so those sections are excluded (eval/eval_digrec.py --exclude-author ...).

HNC Modern Greek

eval_hnc.py evaluates against the HNC Golden Corpus (88K tokens of gold-standard Modern Greek from CLARIN:EL).

How It Works

Pipeline overview

Layer	Speed	Coverage	Source
Lookup table	hash lookup O(1)	14.5M known forms	Wiktionary + LSJ + Sophocles + GLAUx + treebanks
Normalizer	k candidates O(k)	Byzantine orthographic variants	Rule-based candidate generation
Elision expansion	v=7 vowels O(v)	AG elided forms	Vowel expansion against lookup
Crasis table	hash lookup O(1)	~40 common crasis forms	Hand-curated
Particle suffix stripping	suffix check O(1)	AG enclitic forms (-per, -ge, -de, deictic -i)	Strip suffix, re-lookup base form
Verb morphology stripping	prefix check O(1)	Unseen augmented/reduplicated verb forms	Strip augment/reduplication, re-lookup
Dialect normalization	k candidates O(k)	Ionic, Doric, Aeolic, Koine dialect forms	Map dialect forms to Attic equivalents
Compound decomposition	n=word length O(n)	Byzantine compound words	Split at linking vowel, look up base
Spelling correction	BK-tree O(d·m)	ED0-2 suggestions for unknown words	Accent-stripped edit distance
Transformer	beam search O(b·n²)	generalizes to unseen forms	Trained on Wiktionary pairs

Lookup table

The lookup table combines forms from multiple sources:

Source	Forms	Notes
Wiktionary (EN + EL, all periods)	5.2M	Baseline from kaikki.org dumps
LSJ + Sophocles expansion	6.4M net AG additions	Current Wiktionary-Lua expansion plus the marked surface forms from the revision-pinned historical expansion delta; 32K LSJ nouns, 22K verbs, 14K adjectives, and 13.5K Sophocles nouns + 4.6K verbs
GLAUx (Keersmaekers, 2021)	557K	17M-token corpus, 8th c. BC - 4th c. AD, 98.8% lemma accuracy
Diorisis (Vatri & McGillivray, 2018)	76K new	10M-token corpus, Homer - 5th c. AD, 91.4% lemma accuracy. Low-priority pairs (only added when no conflict with existing sources). Contributes to the merged 94.0M-token corpus_freq.json alongside GLAUx, PTA, and the Open Greek Corpus open-text rollup.
HNC Golden Corpus (CLARIN:EL)	1K new	88K-token gold-standard MG corpus, 11K unique form-lemma pairs. Low priority (only added when not in Wiktionary). Also used for MG evaluation.
Perseus / AGDT (CC BY-SA 3.0 US)	81K	The 33 Greek AGDT works: Sophocles, Aeschylus, Homer, Hesiod, Herodotus, Thucydides, Plutarch, Polybius, Athenaeus. Sourced from the original, not the NonCommercial UD release.
DGE (Diccionario Griego-Espanol)	52K	Headword filter coverage for spell-check
LGPN (Lexicon of Greek Personal Names)	44K	Proper noun headword coverage
Perseus Digital Library (L&S, Pape, Bailly, etc.)	176K	Headword filter from multiple classical lexica
VLG (Vocabolario della Lingua Greca, Montanari)	133K	Headword filter coverage; 12K new beyond LSJ/DGE/Perseus
Words in Progress (Aristarchus supplementary lexicon, Montanari & Perrone)	4K	Curated new/rare AG headwords with morphology; ~1,135 new
Closed-class fixes	~500	Articles, pronouns, prepositions mapped to canonical lemmas

Wiktionary Lua expansion. Most of the non-Wiktionary forms above come from running Wiktionary's own grc-decl and grc-conj Lua modules over LSJ and Sophocles headwords, via wikitextprocessor. These are the same modules Wiktionary editors invoke through {{grc-decl}} / {{grc-conj}} templates when they paste in a headword plus its grammatical metadata; the modules emit the full noun, verb, or adjective paradigm. Wiktionary itself only renders those paradigms for headwords an editor has manually written up. Running the modules over the 32K nouns + 22K verbs + 14K adjectives in LSJ and the 13.5K nouns + 4.6K verbs in Sophocles produces millions of inflected forms for classical and Byzantine vocabulary that no editor has touched, which is what lifts the lookup table from 5.3M Wiktionary-only forms to 14.5M total and is why rare-vocabulary coverage on classical and Patristic texts is competitive with rule-based morphological analyzers. Cunliffe's Homeric Lexicon (~12K headwords) isn't expanded this way because its headwords are a subset of LSJ and already covered by the LSJ expansion plus the GLAUx Homeric corpus (557K pairs). See LSJ/Sophocles expansion for the build step.

Verb expansion rejects source entries explicitly identified as prefixes, imperative forms of another verb, or defective verbs whose entry explicitly limits the available persons. Those restrictions preserve source-attested forms without inventing an unrestricted paradigm. It checks the original headword spelling before quantity marks are removed, so the prefix ἡμῐ cannot receive a verb paradigm merely because its letters end in -μι. Ordinary verb entries that describe their conjugation remain eligible.

Synthesized uncontracted active imperatives receive recessive accent from the complete form, rather than inheriting the citation or future stem's accent. For example, ἀκούω and ἀκούσω give ἄκουε and ἄκουσον. The independently documented futures of ἁνδάνω and ἐνίπτω do not supply an aorist stem: their source aorists use different stems. Their observed aorist cells and future participles remain available; irregular aorist parts are not reinterpreted through regular sigmatic participle templates. Explicit vowel lengths in independent Wiktionary verb citations and first-person indicatives are retained in grc_lexical_heads.json. Synthesis uses matching principal-part spellings to recover these lengths without copying their accent. Conflicting lengths remain separate source readings and supply no inferred length. Nominal headers and contradictory grammatical tags cannot supply verb quantities. A declared verb-form page needs a separate lexical definition; quantity notation is removed after accent placement.

The verb expansion is more than just the present system. For each verb headword, an LSJ-entry parser (build/lsj_principal_parts.py) walks the head paragraph and pulls out whatever principal parts LSJ has labeled (fut., aor. 1 / 2, pf., pf. m./p., aor. p., impf., plpf.), feeding each into Module:grc-conj as a positional argument. A principal part must carry a tonal accent. The parser skips an unaccented token when an intact accented form immediately follows in the same clause; it never supplies a missing accent to the source text. About 712 LSJ-only verbs receive at least one parsed principal part this way, lifting them from present-tense-only expansion to full paradigm coverage and adding ~+55 unique forms per eligible verb on average. The classifier itself (build/expand_lsj.py::_classify_verb) routes athematic and irregular verbs explicitly: -ννυμι / -νυμι / -ημι / -ωμι / -αμι / -μι suffix dispatch for the regular cases, plus a hand-coded Attic core paradigm for εἰμί, εἶμι, οἶδα, χρή, φημί and their preverbed compounds (πάρειμι, σύνειμι, εἴσειμι, ...) joined via a backtracking preverb splitter. Across the 28,745-candidate LSJ verb set the full pipeline succeeds on 28,535 (99.27%); the residual failures are all OCR-corrupt or non-Attic dialect entries with no canonical Attic base.

The lookup table is built from Wiktionary kaikki dumps (EN and EL editions for MG and AG, plus EL Medieval Greek), expanded with inflected forms from LSJ (via Wiktionary Lua modules) and the Sophocles lexicon of Roman and Byzantine Greek, then augmented with form-lemma pairs from gold-standard treebanks (AGDT). Each form is indexed under its original, monotonic, and accent-stripped variants, so θεοὶ (polytonic with grave), θεοί (monotonic with acute), and θεοι (stripped) all resolve to θεός. Input can be polytonic, monotonic, or unaccented. AG forms take priority over MG, ensuring classical lemma forms (βιβλίον, φύσις, θεῖος) are preferred over their MG equivalents (βιβλίο, φύση, θείο). Medieval Wiktionary entries are merged into the MG table at build time. When lang="el" is used, 150K MG-specific entries override the AG-first defaults with MG lemma forms (ο instead of ὁ, είμαι instead of εἰμί). For polytonic input (breathings/circumflex), an additional AG-only lookup pass runs first.

When the transformer handles an unseen form, beam search generates multiple candidates and picks the first that matches a known headword from the citation-safe filter. In default mode that includes validated lookup self-maps plus independent Ancient/Byzantine lexicon headwords; with citation_policy="strict_ag" it uses only the independent headword inventory. If nothing matches, the input is returned unchanged. (DGE, LGPN, and the Perseus Digital Library lexica feed the build-time lemma-validation and spell-check filters, not this runtime output filter.)

Wiktionary as upstream: Because Dilemma's lookup tables are built directly from Wiktionary, any missing or incorrect lemmatization can often be fixed by editing the Wiktionary entry itself. When the kaikki dumps are next regenerated and build_data.py re-run, the fix flows into Dilemma automatically. This means the coverage and accuracy of Dilemma improve over time as Wiktionary's Greek coverage improves, without any changes to Dilemma's code.

Rule-based fallback layer

The rule-based morphological analysis fills the gap between the lookup table and the transformer:

Particle stripping (-περ, -γε, -δε, -ι): These are appended particles that create forms the lookup table may never have seen. ὅσπερ is not a separate word in Wiktionary - it is ὅς + -περ. The lookup table would need to store every word x every enclitic particle combination. Stripping is simpler.

Augment/reduplication stripping: A rare verb's aorist (e.g., ἐμόρμυρσεν) might not appear in any corpus or Wiktionary table, but the present stem μορμύρω is there. Stripping the augment and tense markers recovers the connection. The transformer might learn this pattern, but an explicit rule is more reliable for rare verbs it has never trained on. Transformed readings also consult grammatical evidence. An augment or reduplication cannot use a noun homograph as a verb stem, and compound reconstruction retains the supported reading of its base. Orthographic normalization preserves ordinary unambiguous matches while withholding a reading contradicted by the source form's grammar.

Elision/crasis: δ' needs to expand to δέ, κἀγώ needs to decompose to καὶ ἐγώ. These are mechanical text artifacts, not morphology - the transformer would waste capacity learning them.

In short: the rules handle systematic, predictable transformations that the lookup table cannot enumerate exhaustively and the transformer might not generalize to for rare forms. They are the cheap, reliable middle layer between brute-force lookup and expensive ML inference.

Dialect normalization

For Ancient Greek dialect texts (Herodotus, Pindar, Sappho, etc.), the normalizer maps dialect-specific forms to their Attic equivalents so the Attic-heavy lookup table can match them.

Ionic (highest coverage): η/ᾱ alternation after ε, ι, ρ (ἱστορίης → ἱστορίας), uncontracted vowels (ποιέειν → ποιεῖν, τιμέω → τιμῶ), κ/π interrogative interchange (κῶς → πῶς, ὅκου → ὅπου), σσ/ττ alternation (θάλασσα → θάλαττα), ρσ/ρρ alternation (θάρσος → θάρρος), and common word mappings (μοῦνος → μόνος, ξεῖνος → ξένος, κεῖνος → ἐκεῖνος).

Doric: ᾱ/η alternation (Ἀθάνα → Ἀθήνη), word mappings (ποτί → πρός, τύ → σύ), Doric futures (-σέω → -σω).

Aeolic: smooth breathing mapped back to Attic rough breathing (Aeolic systematically drops the rough breathing, a feature known as psilosis).

Koine: σσ/ττ alternation (overlaps with the Ionic rule above and with the period-specific normalization rules in the orthographic normalizer).

d = Dilemma(dialect="ionic")                              # Ionic texts
d = Dilemma(dialect="doric")                              # Doric texts
d = Dilemma(dialect="auto")                               # try all dialects
d = Dilemma(dialect="ionic", period="hellenistic")        # combined

Dialects can be combined with period profiles. Setting dialect implicitly enables the normalizer (no need for normalize=True).

Orthographic normalizer

For texts with non-standard spelling, Dilemma includes an optional orthographic normalizer that generates candidate normalized forms before lookup. This handles:

• Itacism: η/ει/οι/υ all pronounced [i] and interchanged by scribes
• αι/ε merger: αι pronounced [e] and confused with ε
• ο/ω confusion: loss of vowel length distinction
• Missing iota subscripta: ᾳ/ῃ/ῳ written as α/η/ω
• Spirantization: β/υ interchange, φ/π, θ/τ, χ/κ confusion
• Geminate simplification: λλ→λ, νν→ν, etc.

Period-specific profiles (hellenistic, late_antique, byzantine) weight rules by historical probability.

d = Dilemma(normalize=True, period="byzantine")

Transformer model

The transformer is what catches everything the lookup and rule layers miss. It's a 4M-parameter character-level encoder-decoder, the standard SIGMORPHON inflection architecture pointed at lemmatization instead.

It's supervised, not self-supervised. Training data is 3.5M explicit (form, lemma) pairs drawn from Wiktionary inflection tables (AG, MG, and Medieval, including the LSJ verb-paradigm Lua expansion) plus the GLAUx morphologically-tagged corpus. Every example has a known-correct answer. (The LSJ/Sophocles noun expansion and the other treebanks -- Perseus, Diorisis, HNC -- feed the lookup table, not the model; the Gorman treebanks are held-out gold and DiGreC is evaluation-only.)

The vocabulary is character-level, not subword. The encoder reads one Greek character at a time over a ~381-token vocabulary covering monotonic, polytonic, and extended Unicode ranges. A 10-character Greek word is 10 encoder steps, not the ~20 UTF-8 byte steps ByT5 would need, so the model can use a small hidden size without losing morphological resolution.

The model is trained from scratch, with no pretrained weights. End-to-end training on a single GPU takes ~35 minutes (RTX 4090) or ~95 minutes (RTX 2080 Ti). Fine-tuning ByT5-small on the same task takes ~20 hours and ships 300M parameters with the transformers stack attached.

The encoder is shared with three auxiliary classification heads (POS, nominal morphology, verbal morphology), which improve representations via multi-task learning and produce morphological tags as a free byproduct. The same model handles AG, MG, and Medieval data, so AG augment patterns (ἔλυσε → λύω) and MG stem transformations (σκότωσε → σκοτώνω) inform each other; Katharevousa hybrids like εσκότωσε have both signals to draw from.

At inference time the decoder runs beam search and the first candidate that matches the citation-safe headword filter wins. If nothing matches, the input is returned unchanged rather than guessed at. Inference uses ONNX Runtime (~50 MB) by default, with PyTorch as an optional alternative; both backends produce identical outputs.

API Reference

Language and convention options

from dilemma import Dilemma

d = Dilemma()                                  # all periods (default)
d_mg = Dilemma(lang="el")                     # MG only
d_grc = Dilemma(lang="grc")                   # AG only

# LSJ lemma convention
d_lsj = Dilemma(convention="lsj")

# Cunliffe convention
d_cun = Dilemma(convention="cunliffe")

# Triantafyllidis convention (recommended for MG)
d_mg = Dilemma(convention="triantafyllidis")

# Strict AG citation validation for audit/dictionary surfaces
d_audit = Dilemma(lang="grc", citation_policy="strict_ag")

Verbose mode

For ambiguous forms, lemmatize_verbose returns all candidates with metadata so downstream tools can disambiguate using context:

from dilemma import Dilemma

d = Dilemma()

# Proper noun vs common noun: Ἔρις (goddess) vs ἔρις (strife)
candidates = d.lemmatize_verbose("ἔριδι")
for c in candidates:
    print(f"{c.lemma:10s} lang={c.lang} proper={c.proper} via={c.via}")
# Ἔρις       lang=grc proper=True  via=exact

# Multiple candidates: common noun + proper-noun capitalization variant
candidates = d.lemmatize_verbose("πόλεμο")
# -> [LemmaCandidate(lemma="πόλεμος", lang="grc", via="exact"),
#     LemmaCandidate(lemma="Πόλεμος", lang="grc", via="exact+case_alt")]

# Elision with multiple valid expansions
candidates = d.lemmatize_verbose("δ̓")
# -> [LemmaCandidate(lemma="δέ", source="elision", via="elision:ε"),
#     LemmaCandidate(lemma="δή", source="elision", via="elision:η"), ...]

Article-agreement disambiguation: When multiple candidates exist, pass the preceding word to rank by gender/number agreement with a Greek article:

# Prefer candidates matching masculine article τόν
candidates = d.lemmatize_verbose("λόγου", prev_word="τοῦ")
# -> masculine λόγος ranked before proper Λόγος

This only re-ranks candidates, never excludes them. If the preceding word is not a recognized article form, it has no effect.

Each LemmaCandidate has:

• lemma - the lemma string
• lang - "el" (MG, including medieval), "grc" (AG), "med" (medieval provenance label in output)
• proper - True if lemma is a proper noun (capitalized headword)
• source - "lookup", "elision", "crasis", "particle_strip", "verb_morphology", "compound", "article", "normalize", "byzantine_norm", "prefix_strip", "model", "identity", "nonlexical"
• via - how it matched: "exact", "lower", "elision:ε", "suffix_strip", "augment_strip", "θεο+φθόγγος", "+case_alt", etc. For a non-lexical token, the non-lexical class label.
• score - 1.0 for lookup, 0.5 for model, 0.0 for identity fallback
• tag - "X" for a non-lexical token, "" otherwise
• citation - citation-form validation note such as "trusted_headword", "grave_to_trusted_headword", "modern_greek", or "unchanged"
• is_lexical - False for a non-lexical token (source == "nonlexical"), True otherwise

Non-lexical tokens

Real corpora - especially OCR'd lexica, scholia, and the Patrologia Graeca - are full of tokens that are not words: γράφεται variant marks (γρ), Greek numerals (κζ', ,αφ'), editorial references ([76], [49-59]), Latin or citation abbreviations (fr., Herod.), lone punctuation/sigla, and vowel-less consonant fragments. These have no lemma. Dilemma classifies them structurally (pure stdlib, no model) so they can be told apart from a real word that failed to lemmatize, and are never sent to the transformer fallback (which would only manufacture a spurious lemma):

from dilemma import Dilemma, classify_nonlexical, is_lexical

classify_nonlexical("γρ")     # -> "variant-mark"
classify_nonlexical("κζ'")    # -> "numeral"
classify_nonlexical("[76]")   # -> "bracket-ref"
classify_nonlexical("fr.")    # -> "abbreviation"
classify_nonlexical("πλ")     # -> "consonant-cluster"
classify_nonlexical("λόγος")  # -> None  (a real word)
is_lexical("λόγος")           # -> True
is_lexical("γρ")              # -> False

d = Dilemma()
d.lemmatize("γρ")             # -> "γρ"  (returned unchanged, not sent to the model)
d.lemmatize_verbose("γρ")     # -> [LemmaCandidate(source="nonlexical", via="variant-mark", tag="X")]

A census-style consumer filters non-words out of a failure count with d.is_lexical(token) (or the module-level is_lexical). The class labels are dilemma.nonlexical.NONLEXICAL_CLASSES. The classifier is conservative: an elided monosyllable (δ᾿), a word that merely resembles a numeral (τε), and any accented real word stay lexical.

Batch processing

When processing a large corpus (thousands of words), call preload() to enable query-level caching on the SQLite lookup tables. This avoids repeated SQLite round trips for forms that appear multiple times:

d = Dilemma()
d.preload()  # enable query cache - ~40x faster for repeated lookups

for word in corpus:
    d.lemmatize_verbose(word)  # second lookup of same form is instant

preload() is safe to call multiple times (idempotent) and does not change output - it only affects performance. It caches query results on demand rather than loading the full 14.5M-entry table into memory.

POS-aware disambiguation

When a POS tagger (e.g. Dilemma's POS tagger module) provides UPOS tags, lemmatize_pos uses POS to disambiguate between multiple candidates from the regular lookup:

d = Dilemma()
d.lemmatize_pos("αὐτοῦ", "ADV")    # "αὐτοῦ" (adverb: here/there)
d.lemmatize_pos("αὐτοῦ", "PRON")   # "αὐτός" (pronoun: genitive)
d.lemmatize_pos("λευκόν", "NOUN")  # "λευκόν" (noun: white-of-egg)
d.lemmatize_pos("λευκόν", "ADJ")   # "λευκός" (adjective: white)

Curated POS entries can select among regular lookup candidates or override a single candidate when the dictionaries and corpus evidence support a different lemma for that part of speech. An explicit verb tag also overrides a single competing reading represented by lowercase and capitalized candidates, such as Λάμπε selecting λάμπω instead of the independent name Λάμπος. Untagged name readings remain available.

Batch POS calls apply the same overrides when all candidates are case or accent variants of the input. Agreement with a self-map still compares the available citation spellings. If no POS entry matches, case twins retain the scalar API's candidate order and capitalization rule; a single accent variant retains its scalar citation spelling. This fallback checks only structured readings, so unresolved words keep their batched model inference.

With convention="triantafyllidis" or lang="el", POS tags also fix MG self-map issues for adjective and verb inflections. MG lookup tables sometimes return self-maps for inflected forms (e.g. ανθρώπινα maps to itself instead of ανθρώπινος). When POS is ADJ, the masculine nominative citation form (-ος, -ής, -ύς) is preferred. When POS is VERB, the infinitive/1sg form (-ω, -ώ, -μαι) is preferred. Adverbs and nouns keep their MG self-maps unchanged.

The POS lookup tables (~1.82M AG-only entries, ~1.86M combined) are built from six sources in priority order: AGDT originals and licensed corpus votes, LSJ9 indeclinables (2.2K adverbs, prepositions, conjunctions, particles, interjections with unambiguous POS), GLAUx corpus (8.7K entries), MG Wiktionary, AG Wiktionary, LSJ9 grammar. For polytonic input (breathing marks, circumflex), the AG-only POS entries are checked first to avoid MG lemma overrides on Ancient Greek text, mirroring the main lookup's AG-first logic.

Rebuild the corpus POS table after a source refresh with python build/build_treebank_pos_lookup.py --kaikki PATH_TO_KAIKKI. Corpus lemma targets must occur in the merged grc_lexical_heads.json inventory; membership in the generated flat lookup alone is insufficient. The builder requires that inventory before writing outputs and records rejected targets, source fingerprints and all tied owner votes in data/treebank_pos_lookup.audit.json. Use --audit-output PATH to choose another audit destination. A tied owner does not enter the scalar POS table, and rejected source claims still protect against conflicting projections. Explicit Ancient Greek POS calls also reject those documented unsupported targets if an older POS table or the ordinary lookup offers them as a fallback. This applies equally to scalar and batch calls; it does not change untyped lemmatize() or Modern Greek behavior. The rejection audit must match the exact grc_lexical_heads.json bytes. Missing or mismatched evidence raises a data-mismatch error rather than trusting an unbound rejection list. Refresh both files with python -m dilemma download --no-tagger (use --dir PATH for a custom data directory), or rebuild the corpus POS table against the current merged lexical-head inventory. Validation is cached for each model. When a POS entry supplies a distinct supported citation spelling for an inflection, it takes precedence over an accent-stripped lookup collision. An exact independent lookup entry keeps its own spelling, such as οὖ beside the differently accented adverb οὐ. The audit also retains every source-supported capitalized name and ethnic adjective owner before the runtime table's ambiguity filter. The dictionary export combines those owners with exact cased corpus counts and lexical-head evidence, preserving attested names hidden by a flat lowercase owner. Capitalization checks recognize both Unicode uppercase and titlecase letters, including Greek capitals with a written iota such as ᾍ in ᾍδης, without changing their spelling or treating lowercase iota-subscript letters as capitals. A corpus vote that agrees with the bare lookup can still be needed when a lower-priority POS table, a projected spelling, or a capitalized scalar query would select another lemma. A case variant already corrected by the effective POS lookup does not authorize a redundant recovery. These additional entries require at least two weighted corpus votes, a unique highest-weight corpus owner, and an independent definition of the exact lemma and part of speech in the English or Greek Ancient Greek extracts. Tied corpus owners remain in the source evidence and cannot authorize an additional scalar override. Without --kaikki, this additional recovery is disabled. Lowercase and monotonic aliases retain each unambiguous part of speech, and withhold a part of speech with competing owners. Newly recovered entries also account for conflicting corpus claims that did not qualify for recovery. Exact spellings remain available. An exact entry overrides a projected reading only for its own part of speech; an exact noun does not suppress a projected adjective with independent adjective evidence. Adding another POS beside an exact spelling requires independent lexical evidence for that POS, so inherited corpus-tag errors do not introduce unsupported targets. Fresh Wiktionary definitions qualify; --lsj9-headwords PATH also admits explicit LSJ noun/adjective grammar when the file matches the fingerprint and head membership in the merged lexical evidence. Without either source, the existing exact POS scope is retained.

Explicit DET tags retain the established article lemma (ὁ or ο) when an auxiliary POS table instead returns a noncanonical form. This uses the same article inventory and accent-sensitive normalization as the optional article resolver and applies before candidate selection in both scalar and batch calls; untagged words, pronouns, and other POS tags are unchanged.

Spelling correction

For unknown or misspelled words, suggest_spelling returns candidate corrections from the lookup table ranked by edit distance:

d = Dilemma()
d.suggest_spelling("θεός")       # [("θεός", 0), ...]  (exact match)
d.suggest_spelling("θεος")       # [("θεος", 0), ("θεός", 0), ...]  (diacritic error = free)
d.suggest_spelling("θδός")       # [("θεός", 1), ...]  (letter-level ED1)

The approach works in two layers. First, diacritics are stripped from both the input and the dictionary, collapsing the 14.5M-entry lookup into 6.97M unique base forms. ED0/ED1/ED2 matches are found on these stripped forms, then expanded back to their original polytonic variants and ranked by true Levenshtein distance. This means accent and breathing errors (wrong accent, missing breathing mark) are corrected for free, while letter-level errors (θ/δ, ρ/ν) use standard edit distance. The spell index is built lazily on first call.

By default, suggestions include all forms in the lookup table (inflected forms and lemmata from all sources). Two filtering options reduce false positives when resolving to a specific dictionary:

# Only return known LSJ headwords (strictest - ~119K entries)
d.suggest_spelling("ἀγωνιστήριον", max_distance=1, headwords_only="lsj")

# Only return lemmata/citation forms (less strict - ~700K entries)
d.suggest_spelling("ἀγωνιστήριον", max_distance=1, lemmata_only=True)

You can also check headword membership directly:

d.is_headword("θεός")              # True  (LSJ headword)
d.is_headword("θεοί")              # False (inflected form, not a headword)
d.is_headword("θεός", "cunliffe")  # check against Cunliffe headwords

Paradigm generation

dilemma.paradigm resolves Ancient Greek inflection cells by source precedence (jtauber > Morpheus > dilemma_corpus > template), with a small template fallback for the simplest regular cases. Useful when a build pipeline needs full inflection paradigms for every lemma but the kaikki / corpus extracts only carry the cells that happened to be attested.

from dilemma.paradigm import generate, generate_paradigm, ParadigmSlot

slot = ParadigmSlot.verb_finite(
    voice="active", tense="aorist", mood="indicative",
    person="1", number="sg",
)
generate("γράφω", slot)
# ParadigmForm(form="ἔγραψα", source="jtauber")

generate_paradigm("γράφω", "verb")
# {"active_aorist_indicative_1sg": ParadigmForm(...), ...}

Each form is wrapped in ParadigmForm(form, source) so callers can decide whether to trust a cell. Corpus and paradigm-table sources are safe to ship as-is; template fallbacks are best-effort. Inflection keys match the canonical shapes:

Shape	Example
Verb finite	active_aorist_indicative_1sg
Verb infinitive	active_aorist_infinitive
Verb participle	active_aorist_participle_nom_m_sg
Noun	genitive_pl
Adjective	nominative_m_sg

The orchestrator reads paradigm JSONs from $DILEMMA_PARADIGM_DATA (set this to a directory containing jtauber_ag_paradigms.json, ag_verb_paradigms.json, ag_noun_paradigms.json, dilemma_ag_verb_paradigms.json, and/or dilemma_ag_noun_paradigms.json). Missing files yield empty per-source dicts; the orchestrator falls through to whichever sources are available and finally to the template fallback.

Templates handle thematic -ω verbs (present-system active only), vowel-stem 1st / 2nd-declension nouns, and three-termination -ος / -η / -ον adjectives. Anything that needs accent re-placement or stem allomorphy (contract -άω / -έω / -όω, athematic μι-verbs, suppletive verbs, 3rd-decl consonant stems) returns None so callers can leave the cell empty rather than ship a bogus form. Pass allow_template=False to skip the template path entirely and only return cells with explicit source attribution.

A CLI entry point fills missing inflection cells across a build's canonical hash, mapping {filepath: entry}:

python -m dilemma paradigm fill --in pending.json --out filled.json [--with-templates]

Each filled cell is recorded in a sidecar inflections_source.<dialect>.<key> field on the entry so downstream consumers can render generator-filled cells distinctly from corpus-attested ones.

Elision expansion

Ancient Greek texts frequently elide final vowels before a following vowel, marking the elision with an apostrophe (U+0313 in polytonic encoding, U+1FBD/U+02B9/U+02BC/U+1FBF/U+2019/ASCII '/U+0060 in other encodings). All of these resolve identically: the lemma never depends on which apostrophe codepoint the source text used.

Exact elided entries first: the lookup table stores corpus-derived lemmas for elided forms under one canonical key (U+1FBD), and these win before any expansion heuristic: ὅτ᾽ -> ὅτε (not the more frequent homograph ὅτι, which never elides in Homer), μ᾽ -> ἐγώ, σ᾽ -> σύ, κ᾽ -> ἄν, θ᾽ -> τε, ποτ᾽ -> ποτέ, αὖτ᾽ -> αὖτε. One residual static-lookup limit: a few Homeric ὅθ᾽ tokens are the locative ὅθι rather than temporal ὅτε, which only syntactic context can settle.

Verbose and POS-aware calls prioritize an exact entry over reconstructed expansions when its original lemma is supported by the merged lexical inventory for the requested language. This checks the original citation before convention remapping, including proper names. Unsupported exact targets retain their former ordering, and every alternative remains available. A matching POS reading can still select another candidate.

Forms without an exact entry fall through to the expander, which strips the elision mark and tries each Greek vowel against the lookup table:

Elided	Expanded	Lemma
ἀλλ̓	ἀλλά	ἀλλά
δ̓	δέ	δέ
τ̓	τε	τε
ἐπ̓	ἐπί	ἐπί
ἔφατ̓	ἔφατο	φημί
κατ̓	κατά	κατά
καθ᾿	κατά	κατά
ἀφ᾿	ἀπό	ἀπό
βάλλ̓	βάλλε	βάλλω

Consonant de-assimilation: Before rough breathing, Greek assimilates voiceless stops to aspirates (τ->θ, π->φ, κ->χ). The elision expander reverses this: καθ᾿ tries both καθ- and κατ-, ἀφ᾿ tries both ἀφ- and ἀπ-, recovering prepositions like κατά and ἀπό.

Frequency ranking: When multiple expansions match the lookup table, candidates are ranked by lemma-keyed corpus attestation (see attestation() below), so common prepositions like κατά always beat obscure verbs like κάθω. Function words are further prioritized when the stem matches a known elision pattern, and proper nouns are deprioritized.

Polytonic input automatically restricts expansion to the AG lookup table, avoiding false matches from MG monotonic forms.

Corpus attestation

Dilemma.attestation(lemma) returns a per-lemma diachronic profile built from the GLAUx and Diorisis corpora plus the OGA and Pedalion corpora consumed via the Open Greek Corpus standardized annotation export (34M deduped tokens, 304K lemmas): how often, when, where, and in what genre a lemma is attested. It is keyed by the corpora's own NFC polytonic lemma annotation and returns None for an unattested lemma.

d.attestation("ἀείδω")
# {
#   "total": 1942,                                          # deduped frequency
#   "source_counts": {"glaux": 1869, "diorisis": 1240},    # independent, NOT summed
#   "by_genre":   {"philosophy": 367, "poetry": 441, "history": 263, ...},
#   "by_century": {"-8": 49, "-7": 20, "-6": 40, ...},      # signed; -8 = 8th c. BC
#   "by_dialect": {"Aeolic": 2, "Attic": 249, "Attic/Koine": 662, ...},  # GLAUx only
#   "dominant_pos": "verb",
# }

The artifact is data/lemma_attestation.json, built by the standalone build/build_lemma_attestation.py pass. The three sources annotate many of the same texts, so the frequency (total and the by_* breakdowns) is deduplicated at the work level by TLG id (each work counted once, with priority GLAUx > Diorisis > OGA > Pedalion; GLAUx preferred for its dialect and richer metadata) - a union, not a sum. OGA is all-automatic annotation (a form of model output), so it is the lowest-priority source; its ~17M tokens cover 795 works - the Suda, the Iliad scholia, Eusebius, patristic homilies - that no hand-checked treebank annotates, and works the Gorman treebanks annotate are excluded from it entirely (Gorman is the held-out gold). Pedalion (manual-gold AGDT trees, PROIEL already dropped by cog) adds ~39K deduped tokens from works no other source covers (Sextus Empiricus, Menander, papyri, poetic fragments); its Gorman-provenance rows are skipped and its constructed example-sentence collections feed only lookup/POS, not attestation. source_counts separately keeps each lemmatizer's full independent count (overlapping, never summed): agreement is a confidence signal, and a lemma seen only in a non-preferred source's reading of a shared work has total 0 but a non-empty source_counts. It is distinct from the form-keyed, genre-only corpus_freq.json: this one is lemma-keyed and adds the century and dialect axes. Because the corpora are not sense-disambiguated there is one profile per lemma string, so dominant_pos disambiguates a noun-vs-verb homograph but not same-POS senses. The file's _meta block documents the full schema, the signed-century scheme, and the caveats.

Attested forms: the "attested only" gate and form attestation

The lemma profile above has a surface-FORM sibling: which exact inflected forms actually occur in the corpus, with their own usage distribution and the passages that attest them. Dilemma.form_attestation(form) returns it, or None if the form is unattested:

rec = d.form_attestation("μῆνιν")
# {
#   "total_count": 349, "n_works": 128,
#   "source_counts": {"diorisis": 104, "first1k": 219, "glaux": 128, "pg": 28, "pta": 4},
#   "by_century": {-8: 14, -6: 5, -5: 12, ...},     # usage-by-year axis
#   "by_genre":   {"poetry": 40, "history": 34, "philosophy": 13, ...},
#   "by_century_genre": {-8: {"poetry": 14}, ...},  # joint, for a usage heatmap
#   "dominant_pos": "other",   # untagged raw-text sources (first1k/pta/pg) -> "other"
#   "citations": [                                 # example passages (work + locus)
#     {"author": "Homerus", "title": "Ilias", "source": "glaux",
#      "century": -8, "locus": "1.1", "locus_scheme": "line", "count": 1}, ...]
# }

Both directions of the lemmatizer/generator take an attested_only flag (off by default, so nothing changes unless you ask for it):

# Input: only lemmatize forms that actually occur in the corpus (exact NFC,
# elision resolved); an unattested form returns None instead of a guess.
d.lemmatize("μῆνιν", attested_only=True)          # "μῆνις"
d.lemmatize("υπολογιστές", attested_only=True)    # None (modern, unattested)
d.lemmatize_batch(words, attested_only=True)      # None at unattested positions
# attach the citations to each candidate:
d.lemmatize_verbose("μῆνιν", with_attestation=True)

# Output: only generate paradigm cells whose form is attested (grave/case
# folded, since generated forms are citation-style), with optional citations.
from dilemma.paradigm import generate, generate_paradigm, ParadigmSlot
generate_paradigm("λύω", "verb", attested_only=True, with_attestation=True)

The data lives in two SQLite artifacts built by the standalone build/build_form_attestation.py pass. It reads the lemmatized GLAUx + Diorisis treebanks plus the raw-text First1KGreek, PTA, Patrologia Graeca and byzantine-vernacular corpora (for late-antique, patristic and Byzantine coverage), deduping each work once by source priority (so total_count is a deduped union while citations keep every source's passages). The Patrologia Graeca and byzantine-vernacular texts are read from the Open Greek Corpus.

The Patrologia Graeca comes in three tiers of evidence, best first, and a Migne volume is counted once under the best tier that covers it:

1. The CC-BY calfa-co text as the corpus serves it, with whole-token OCR corrections applied and each volume carved into its works. 22 volumes, loci PG<volume>.<page>.
2. The corpus's Qwen3.6-27B re-OCR of the public-domain Migne scans, which replaced a retired first-generation OCR pass that misread whole sorts (ϖ as σ). 48 further volumes, loci pg<volume>_<page>.<line> - Eusebius, Athanasius, Basil, both Gregories, Chrysostom, Cyril of Alexandria, Theodoret, John of Damascus.
3. The raw first-generation calfa-co dump, read from <corpus>/sources/cgpg/PG*/PG*_text.txt. It is the only text for 11 volumes nothing else covers, and for the 22 carved volumes it is read as secondary evidence: it credits source_counts and citations for the part of a volume the carve plan leaves out, without touching the deduplicated totals the better text decides.

The raw-text sources (especially the OCR'd Patrologia Graeca) are noisier than the treebanks, so a form attested only there is lower-confidence; source_counts tells you which corpora attest each form. Both artifacts are opt-in downloads, kept out of the base dilemma download:

python -m dilemma download --with-attestation   # form_profile.db: gate + distribution
python -m dilemma download --with-citations      # + form_citations.db: example loci

--with-attestation is enough for the gate and the usage distribution that powers the usage-by-year graph and heatmap; --with-citations adds the example loci (and implies the profile). Without them, attested_only and form_attestation raise a clear "download it" error.

Example loci require matching schema versions, content hashes and source hashes in the two databases. If the citation file belongs to another build or lacks this metadata, form_attestation still returns the usage profile, with an empty citations list and a citations_note asking you to download both files together. max_citations=0 reads only the profile.

Saying "I don't know": guess=False

By default the lemmatizer always answers: when nothing structured resolves a word it falls back to the transformer, and when even that fails it returns the input unchanged. For a dictionary-lookup caller a wrong lemma is worse than no lemma (a guessed headword puts a false definition in front of the reader), and an echoed input is indistinguishable from a successful identity lemmatization. All five lemmatization entry points therefore take guess=False, which skips the transformer and returns None (or an empty candidate list) instead of a guess or an echo:

d.lemmatize("μάκαπ")                         # "μάκαπ" (echo - OCR junk, unresolvable)
d.lemmatize("μάκαπ", guess=False)            # None ("I don't know")
d.lemmatize_batch(words, guess=False)        # None at unresolvable positions
d.lemmatize_verbose("μάκαπ", guess=False)    # [] - no identity/model candidates

Non-Greek and NON-LEXICAL tokens (see classify_nonlexical) still pass through unchanged: those are classifications, not lemma guesses. For a per-candidate confidence signal, use lemmatize_verbose() - each candidate's score (1.0 lookup ... 0.5 model ... 0.0 identity echo) and source say how much to trust it.

Repairing corrupt lemma annotations

Lemma annotations in treebanks and OCR-derived data break in learnable ways: OCR letter confusions (μάκαπ for μάκαρ), digamma noise (ϝέκταρ for νέκταρ, πρόσθεϝ for πρόσθεν), diaeresis/gemination variation (ἐυμελίης for ἐϋμμελίης), and truncation (ἐπιτροχάδη for ἐπιτροχάδην). repair_lemma() validates a lemma against the canonical AG headword inventory (LSJ + Cunliffe, shipped in data/) and repairs a corrupt one - preferring evidence from the surface form it was assigned to, which is usually correct even when the annotation is not:

d.repair_lemma("μάκαρ")                      # "μάκαρ"  (already canonical)
d.repair_lemma("μάκαπ")                      # "μάκαρ"  (edit-distance repair)
d.repair_lemma("ϝέκταρ")                     # "νέκταρ" (digamma normalization)
d.repair_lemma("Πειραί", form="Πειραΐδης")   # "Πειραΐδης" (form-guided)
d.repair_lemma("διαπέταμαι", form="διέπτατο") # "διαπέτομαι"
d.repair_lemma("ξζψχβγ")                     # None - never echoes junk back

The same digamma handling also works on surface forms in plain lemmatization: lemmatize("πρόσθεϝ") -> πρόσθεν, lemmatize("ϝάναξ") -> ἄναξ (a variant is accepted only if it itself resolves in the lookup, so this never invents a lemma).

POS tagger and dependency parser

Dilemma also ships a diachronic Greek POS tagger and dependency parser under dilemma.tagger, on a lightweight onnxruntime + tokenizers inference backend. Ancient (grc) and Medieval/Byzantine (med) use a fine-tuned GreBerta encoder (Apache-2.0) that PRESERVES polytonic accents and breathings; Modern Greek (el) uses a Greek-BERT encoder with a Dozat-Manning biaffine dependency head:

pip install "dilemma-nlp[tagger-onnx]"   # torch-free runtime: onnxruntime + tokenizers
# or "dilemma-nlp[tagger]" for the full tagger (runtime + torch + transformers, for training)
python -m dilemma download               # also fetches the tagger weights

from dilemma import Tagger

tagger = Tagger(lang="grc")    # Ancient Greek (GreBerta, accent-preserving)
results = tagger.tag(["μῆνιν ἄειδε θεὰ Πηληϊάδεω Ἀχιλῆος"])
for tok in results[0]:
    print(tok)
# {'form': 'μῆνιν', 'upos': 'NOUN', 'lemma': 'μῆνις',
#  'feats': {'Case': 'Acc', 'Number': 'Sing', 'Gender': 'Fem'},
#  'head': None, 'deprel': None, 'raw_form': 'μῆνιν'}

Supports lang="el" (Modern Greek), lang="grc" (Ancient), and lang="med" (Medieval/Byzantine). When lemmatize=True (the default), the tagger preloads the lemmatizer internally and returns a lemma field on every token (POS-aware, from the model's predicted UPOS).

Tokenization is whitespace-based with two refinements, so the output can contain more tokens than text.split(): leading/trailing punctuation becomes standalone PUNCT tokens (ἄειδε, -> ἄειδε + ,; elision and aphaeresis apostrophes like δ’ stay attached), and Modern Greek multiword tokens expand (στο -> σ + το).

The Modern Greek (el) model is trained ENTIRELY on openly licensed gold treebanks - UD_Greek-GUD (Standard MG) plus the Cretan, Lesbian, and Messinian dialect treebanks, all CC BY-SA 4.0 - NOT the NonCommercial UD_Greek-GDT. It does UPOS + UD-feature tagging AND dependency parsing (so el tokens carry head/deprel; Modern Greek multiword tokens like στο = σ + το are split automatically). On the held-out GUD (Standard MG) test split:

Modern Greek tagger	UPOS	feats	UAS	LAS
Dilemma el (GUD + CC BY-SA dialects, ~38K tokens)	97.9%	99.4%	89.5%	86.1%
reference: a model trained on the NonCommercial UD_Greek-GDT (~62K tokens)	97.5%	--	90.5%	88.4%

So declining the NonCommercial GDT costs only ~2 LAS (and gains UPOS) on ~40% less data, while keeping the model fully CC-licensed. The dialect treebanks are not optional padding: ablating them collapses parser LAS from 86.1% to 58.9%, because they nearly double the dependency-training signal.

The Ancient and Medieval (grc/med) model now carries a Dozat-Manning biaffine dependency head too, trained on GLAUx with the CC BY-SA AGDT relation set (the NonCommercial PROIEL and UD Perseus are excluded). So grc/med tokens also carry head/deprel (AGDT relations such as PRED, SBJ, OBJ, ATR, ADV, plus the _CO/_AP coordination and apposition variants). On the held-out GLAUx test split:

Ancient Greek tagger	UPOS	feats	UAS	LAS
Dilemma grc (GLAUx + AGDT, ~19M training tokens)	96.3%	99.5%	83.6%	78.3%

(dev-split numbers run higher - UPOS 97.9 / feats 99.7 / UAS 88.6 / LAS 84.3; the test split is a harder held-out sample.) The grc model is Morpheus-free (trained --no-prior), so nothing in the runtime path needs the Morpheus binary.

The tagger is ~25x faster than gr-nlp-toolkit on real-world Greek text after gr-nlp-toolkit's PR #29 (which Dilemma's author contributed; pre-PR the gap was ~215x). On the full Iliad (24 books, 146K tokens) it tags in 19.5 s. Trained weights, treebank sources, and the lang="med" Medieval/Byzantine model are documented in dilemma/tagger/__init__.py.

The ONNX taggers are reproducible from train_tagger.py (Greek-BERT/GreBerta encoder + per-feature heads), export_tagger_onnx.py (3-input ONNX export), and the data builders build/build_tagger_data.py (Ancient Greek from GLAUx) and build/build_mg_tagger_data.py (Modern Greek from UD_Greek-GUD + the CC BY-SA dialect treebanks), with convert_treebank.py doing the AGDT postag→UD mapping. Both are Morpheus-free, so the taggers reproduce from the public repo plus the corpora.

Device and throughput planning (CPU vs GPU)

Two engines sit behind the API and they scale very differently:

• The lemmatizer is the workload: a 14.5M-form SQLite lookup, the rule layers, and a small char-transformer beam search for the tail. It is CPU-bound and, in aggregate, memory-bandwidth-bound. A GPU does not speed it up.
• The tagger (GreBerta ONNX) is a batched encoder forward pass. GPU- acceleratable in principle, but it is a small fraction of the per-token cost and runs fast on CPU, so it is not where a full-corpus pass spends its time.

Verdict: a full-corpus annotation pass is CPU + memory-bandwidth bound, not GPU-bound. Measured directly (open-greek-corpus-annotations/scripts/bench_devices.py) over a fixed ~50k-token Ancient Greek sample on two boxes:

Box	lemma, 1 worker	lemma saturation	tag (CPU, 1 proc)	coupled tag+lemma per shard on GPU
EPYC 7B12 (64c / 2.5-3.3 GHz, DDR4)	54 tok/s	~1,100 tok/s @ ~32 workers	745 tok/s	361 tok/s (12 shards, 90% GPU util) — worse
Threadripper PRO 7995WX (96c/192t, 5.39 GHz, DDR5 8-ch)	110 tok/s	~2,460 tok/s @ ~48-61 workers	2,531 tok/s	not run (proven counterproductive)

The lemmatizer scales with cores until it hits a memory-bandwidth wall, then flattens. On the 7B12, per-worker throughput fell 54 → 38 (16w) → 31 (32w) → 18 (61w), so aggregate peaked near ~1,100 tok/s around 32 workers and more workers bought almost nothing. The 7995WX starts at 2x the per-worker rate and, with DDR5 8-channel bandwidth, pushes the wall out: 110 (1w) → 995 (16w) → 1,728 (32w) → 2,327 (48w) → 2,460 (61w), with per-worker sliding 110 → 62 → 54 → 40 as the wall is approached. Aggregate peaked near ~2,460 tok/s at ~48-61 workers (2.2x the 7B12). Tag-on-CPU also scales with cores (745 tok/s on the 7B12 vs 2,531 on the 7995WX), so it never becomes the bottleneck.

The GPU is nearly irrelevant, and coupling the two engines on it is an anti-pattern. Running the pipeline as N sharded processes on a GPU box gives each shard its own GreBerta CUDA session, so scaling the CPU-lemma parallelism to the core count spawns dozens of CUDA sessions, exhausts VRAM (util collapses toward 0), and the run tops out below the CPU-only rate — it measured 361 tok/s (12 shards maxing a 24 GB card) versus 1,100 tok/s for the same box's CPU lemmatizer. Never run one GPU tagger session per worker.

Recommended shape: decoupled, pure-CPU scale-out. Run N CPU workers that each tokenize + tag + lemmatize (equivalently, many CPU lemmatize workers fed by a few batched CPU tagger processes), tagger on CPU, no per-shard GPU session at all. Set the worker count at the box's memory-bandwidth knee — the point where per-worker throughput has fallen off but aggregate is still near its max (~32 on the 7B12, ~48-64 on the 7995WX). Do not exceed it: past the knee you pay for cache/bandwidth contention with no throughput gain.

Picking hardware for a large pass: buy cores and memory bandwidth, not a GPU. A high-clock, many-DDR5-channel CPU (Threadripper PRO / EPYC Genoa) beats a GPU box for this job, and a low-end GPU (or none) is fine.

Device selection and verification. ONNX sessions pick their device automatically (dilemma/_ort_providers.py): CUDA when onnxruntime-gpu is installed, else CPU; override with DILEMMA_ORT_PROVIDERS (e.g. CUDAExecutionProvider,CPUExecutionProvider to force GPU, or CoreMLExecutionProvider,CPUExecutionProvider for Apple's Neural Engine). A pure-CPU box (plain onnxruntime) correctly reports Tagger.on_gpu=False / Tagger.providers=['CPUExecutionProvider'] and runs at full speed. Watch out for the silent trap: onnxruntime falls back to CPU if onnxruntime-gpu is missing or CUDA/cuDNN mismatches, so if you intend to use a GPU, assert tagger.on_gpu up front (make_session() also warns) rather than discovering a CPU fallback from a slow run.

Rule of thumb, and the one caveat. The saturation numbers above are an upper bound measured on ordinary running text, where the lookup resolves ~95% of tokens and the char model only fires on the tail. Real throughput tracks how often the expensive char-model beam search fires, which is a property of the text: lookup-heavy prose runs near the box's saturation rate, but dense rare-vocabulary material - OCR'd lexica (Suda, Hesychius), scholia, and Patrologia Graeca commentary, where a large share of tokens are unseen headwords/forms - drives the beam search far more and runs several times slower per token. Plan with the mix you actually have. Scaling still comes from cores: because the beam search is compute-bound (not bandwidth-bound like the lookup), a beam-heavy pass keeps scaling with workers past the lookup knee, so use ~1 worker per hardware thread for it. One more structural limit: a single work runs on one worker, so a run's wall-clock is floored by its largest work; split giant lexica/scholia across workers if that tail dominates. ETA ≈ (remaining word tokens) / (measured tok/s on that text) - e.g. the ~45M-token, beam-heavy lexica/scholia tail of the Open Greek Corpus measured ~1,150 tok/s aggregate at 96 pure-CPU workers on the 7995WX and scaled up with more workers (the run used ~160, since a beam-heavy pass keeps scaling past the lookup knee), well under the ~2,460 tok/s the same box hits on ordinary prose.

Greek Coverage

Language codes

Code	Period	ISO standard
el	Modern Greek (including vernacular medieval, Katharevousa, regional)	ISO 639-1
grc	Ancient Greek (Homer through Byzantine literary Greek)	ISO 639-2

Code and API calls use ISO 639 language codes: el for Modern Greek and grc for Ancient Greek. In English text we often use the shorthands MG (Modern Greek) and AG (Ancient Greek).

For Dilemma's purposes, MG (el) includes Katharevousa, even though Katharevousa often benefits from AG lemmatization due to its archaizing vocabulary and morphology. Medieval/Byzantine Greek has two components: vernacular medieval Greek (ancestor of Modern Greek, merged into el) and literary Byzantine Greek (classicizing, Atticist-influenced, resolved via the AG lookup under grc).

For lemmatization, the two-way split works because Byzantine literary Greek is classicizing (handled by grc), while vernacular medieval Greek is the ancestor of Modern Greek (handled by el). The med label still appears in LemmaCandidate.lang for forms from the medieval Wiktionary dump, but these are merged into the el lookup at build time.

Note: Dilemma's POS tagger and dependency parser use lang="grc" for Byzantine text. Byzantine literary syntax (polytonic, full case system, optative mood) is closer to Ancient Greek, so the AG-trained tagger handles it well.

Modern Greek varieties

Variety	Wiktionary-tagged headwords
Standard Modern Greek (SMG/Demotic)	877K entries (core)
Katharevousa	283+ tagged, hundreds more formal/place terms
Cretan	273
Cypriot	199
Heptanesian (Ionian)	18
Maniot	3
Medieval/Byzantine (vernacular)	3K (merged into MG - vernacular medieval is the ancestor of MG; literary Byzantine is Atticist-influenced and resolves via the AG lookup, not this table)

Ancient Greek varieties

Variety	Wiktionary-tagged headwords
Epic/Homeric	3,755
Ionic	1,638
Attic	1,279
Koine	1,209
Byzantine (literary)	496
Doric	456
Aeolic	163
Laconian	52
Boeotian	15
Arcadocypriot	11

The counts above are Wiktionary headwords explicitly labeled with a dialect tag. Each headword generates a full inflection paradigm (10-40 forms for verbs, 4-8 for nouns), so Wiktionary-derived form coverage is much larger than the headword count suggests.

However, Wiktionary tags are only a fraction of Dilemma's actual dialect coverage. Corpus-derived form-lemma pairs add substantially more: GLAUx contributes 76K Ionic pairs from Herodotus and the Hippocratic corpus, with Diorisis and AGDT adding more across Herodotus, Thucydides, Xenophon, Demosthenes, and others. The dialect normalization layer (Ionic, Doric, Aeolic, Koine) then maps remaining dialectal forms to their Attic equivalents for lookup, catching forms that no corpus or dictionary has catalogued.

Katharevousa forms are the primary non-SMG target for Modern Greek - they mix AG morphology (augments, 3rd declension genitives) with MG vocabulary. The strong Epic/Homeric coverage (3,755 tagged headwords plus extensive GLAUx corpus data) is directly relevant for literary texts based on Homer.

Medieval/Byzantine Greek

Medieval/Byzantine Greek has two distinct registers that Dilemma handles differently. Vernacular medieval forms are merged into Modern Greek (el) since they are the direct ancestor of MG. Literary Byzantine forms are classicizing and resolve via the AG (grc) lookup. EL Wiktionary's "Medieval Greek" category (3,158 entries, 3,080 headwords) is roughly 71% vernacular and 29% literary Byzantine, based on presence of polytonic diacritics:

• Vernacular (~71%): δέρνω, θυμώνω, χτενίζω, βρίσκω, γούνα, ναράντζι, βουρκόλακας, ξεχαρβαλώνω - early MG vocabulary
• Literary Byzantine (~29%): ἀποφθέγγομαι, αἰθεροπόρος, περικαλλής, κριθάλευρον - Atticist-influenced forms
• Medieval-specific: μαξιλάριν, ἀδελφάτον, κασσίδιον, ἴνδικτος, γαστάλδος - neither pure AG nor modern MG

Merging all into el works because the AG lookup runs first. The 29% literary forms typically already exist in the AG table and resolve there; only the vernacular and medieval-specific forms actually fall through to the MG lookup. On the DBBE benchmark, only 2 of 8,342 tokens resolved via the medieval table, while 92.8% came from the AG lookup.

Development

Full installation

git clone https://github.com/open-greek/dilemma.git && cd dilemma
pip install -e ".[onnx,lm,torch,dev]"  # build + train + test deps
python build_data.py --download
python build_lookup_db.py              # SQLite for instant startup
python fix_selfmaps.py                 # fixes inflected forms that self-map
python train.py                        # full scale (~45 min on RTX 2080)
python export_onnx.py                  # optional: PyTorch-free inference (also needs: pip install onnx)

Training

1. Build data

Downloads all 5 kaikki dumps and extracts every form-lemma pair from inflection tables. Non-Greek characters are filtered out.

pip install -e ".[lm]"                       # build-data deps (lxml, betacode, beta-code, pygtrie)
python build_data.py --download             # downloads + extracts (~1.5GB total)

2. Train

Trains the character-level transformer on the extracted pairs. Use --scale to control the training size.

python train.py --scale test                # quick sanity check (20K pairs, ~15 sec)
python train.py --scale full                # all data (~45 min on RTX 2080, default)
python train.py                             # same as --scale full

Legacy --scale 1/2/3 flags are still accepted for compatibility.

Training scales

Every scale includes 100% of non-standard varieties (Medieval, Katharevousa, Cypriot, Cretan, Maniot, Heptanesian, archaic, dialectal). The remaining budget is split 50/50 between Ancient Greek and standard MG. Underrepresented tense categories are oversampled to compensate for their rarity in Wiktionary's paradigm tables, following Swaelens et al. (2025)'s finding that perfects are underrepresented in training data relative to Byzantine text. Aorist forms (3x, critical for stem-changing 2nd aorist), perfect (3x), future (3x), imperfect (2x), and pluperfect (5x, rarest at 0.15% of pool) are oversampled proportionally to their rarity and the degree of stem change from the present form.

Scale	Training pairs	Varieties	AG	SMG	Time (RTX 2080)
test	20K	9K (100%)	5.5K	5.5K	~15 sec
full	3.5M (all)	9K (100%)	1.5M (100%)	1.7M (100%)	~95 min

Models save to model/{lang}-test/ (test scale) or model/{lang}/ (full scale).

Eval accuracy is the model's score on held-out pairs without the lookup table. In practice, the lookup resolves most forms instantly and the model only handles truly novel words. When the model is used, beam search generates 4 candidates and the first one that matches a known headword in the lookup wins. If none match, the input is returned unchanged (safe fallback).

Multi-task learning

When training pairs include POS tags (from Wiktionary) and morphological features (from GLAUx), the model jointly predicts POS, nominal morphology (gender/number/case, 45 labels), and verbal morphology (tense/mood/voice, 69 labels) alongside the lemma via auxiliary classification heads on the encoder output. This follows Swaelens et al. (2025)'s finding that multi-task learning (joint POS + morphology + lemma) improved Byzantine Greek lemmatization by ~9 percentage points. Each auxiliary loss is weighted at 0.1x relative to the lemmatization loss. At full scale, the heads reach 90.4% POS, 81.5% nominal, and 91.2% verbal accuracy on the held-out set.

Training uses a linear warmup LR scheduler (500 steps warmup, then linear decay) and gradient clipping (max norm 1.0) for stable convergence.

LSJ/Sophocles expansion

To regenerate the expanded lookup table from LSJ and Sophocles sources:

pip install --force-reinstall --no-deps git+https://github.com/tatuylonen/wikitextprocessor.git
python build/expand_lsj.py --setup           # build Wiktionary Lua module database
python build/expand_lsj.py --expand          # expand LSJ nouns
python build/expand_lsj.py --expand-verbs    # expand LSJ verbs
python build/expand_sophocles.py --expand    # expand Sophocles nouns
python build/expand_sophocles.py --expand-verbs  # expand Sophocles verbs
python overlay_lsj.py                        # recover pinned historical expansion gaps
python build/expand_lbg.py                    # inflect Byzantine headwords (gated gap-fill)

This requires LSJ9 data from lsj9 (included in data/lsjgr_bridges.json and data/lsj9_frequency.json) and the Sophocles TEI data (included in data/sophocles/).

The final overlay is intentional. A May 2026 base-data refresh replaced the 9,947,605-entry expanded AG table with a 2,354,300-entry Wiktionary base, and the June re-expansion recovered only part of the prior paradigms. The overlay derives only the expansion delta from the known-good, revision- and SHA-256-pinned March artifact in data/ag_expansion_reference.json; it never restores a historical base mapping and never overrides a current mapping. It restores marked surface spellings only, not the generated accent-stripped fallback keys that can shadow unrelated analyses sharing the same bare skeleton. The generated data/ag_expansion_exclusions.json also removes historical rows that conflict with the pinned 1.3.1 resolved lookup or with Dilemma's nonlexical classifier. Its generator is build/build_ag_expansion_exclusions.py. Running it without --reference downloads and verifies that exact artifact.

After withholding approximate irregular-compound paradigms, the overlay separately preserves actual source cells and their original, monotonic, and unaccented lookup aliases within the historically represented families. This requires a clean merged Wiktionary, corpus, or reviewed textual cell and a positively defined Ancient lexical owner; a historical letter pattern alone never generates a spelling. Existing independent readings and hard spelling rejections take precedence. Unresolved source or marked-cell ownership ties, and unsupported aorist, perfect, or pluperfect annotations under εἰμί/εἶμι compound owners do not license new aliases without direct ownership evidence. An automatic shorter compound owner is also withheld when only a longer, independently defined compound accounts for the actual source cell's preverb; that comparison never infers a replacement owner. Replacing an unsupported existing target owner requires direct evidence. Missing citation accents may be supplied from a unique positive head. Structurally impossible citation accents (on a consonant, duplicated within a syllable, or an invalid second accent) can use that same unique-head path while preserving breathing, subscript, and case. Actual source spellings remain unchanged in the audit. A valid existing accent is preserved because it can distinguish a different lexeme or contracted form, including a productive enclitic accent. A solely capitalized automatic cell also needs matching lowercase or direct evidence before gaining aliases, and cannot erase a possible existing name. Final elision is normalized to Greek koronis while the original source spelling is retained in the audit. --source-alias-audit FILE records all readings, restorations, canonicalization, ambiguity skips, and portable input/source hashes. The merged source pairs, reviewed exact-form specification, and grc_lexical_heads.json are required inputs for this step.

With --kaikki DIR, the overlay also checks historical-only elisions against both fresh Ancient Greek dumps in en-el/ and el/, and binds noun input validation to that same directory. It can replace an unsupported historical owner only when all positively defined current source readings agree on one PRON or DET owner, and every reading fits an explicit case/number cell in that owner's independently defined entry of the same POS. The complete cell must be the marked stem plus one literal vowel. Source quantity notation and initial capitalization are normalized; accents, breathings, and other letters are preserved. Competing owners or POS readings, incompatible cells, a positive historical marker variant, and a matching complete unelided word all withhold the change. Existing predecessor mappings and later changes remain protected. When rebuilding from a reversed alias audit, an exact historical cell with its unchanged historical owner is checked again against current evidence, so a still-supported repair is reproducible. Independent marker variants remain protected. This check creates no bare lookup aliases.

The historical overlay counts remain unchanged. Proven replacements run after the normal alias recovery, only while the former historical owner still matches, and join the reversible canonicalized audit. The audit retains all source claims, own-entry cells, definitions, and withheld decisions. It pins both fresh dumps and the imported definition classifier and repair helper. Omitting --kaikki disables this additional check; providing it requires both dumps and rejects missing, malformed, or changed input files.

The Ancient export retains independently cited proper-name forms even when the flat lookup assigns their spelling to a common-word lemma. The finite source_forms specifications in data/hunspell_grc_postclassical_review.json record the spelling, owner, citation, and source. They preserve forms such as Πῶλε and Ὀνησίμῳ without exempting unrelated capitalized common nouns from the case filter.

An already reviewed compatibility spelling also survives the case filter when it has positive exact-case corpus evidence or no lowercase spelling survives. This retains inherited capitalized spellings and work titles without inventing a proper-name lemma. The audit distinguishes those two reasons and records the retained spelling, its count, and the existing ownership decision. Unattested lowercase name aliases and redundant sentence capitals remain filtered; explicit spelling, quotation, and bare-fragment rejections still take priority. Restored capitals with unresolved ownership must also pass the structural and corpus respelling checks; exact-case counts alone do not waive those checks.

Exact reviewed forms also retain Archimedes' Doric ἀποχρὴ under ἀπόχρη, with its contextual acute twin. This textual exception does not remove accents from other valid citation heads when recovering aliases.

The Diorisis pair extractor preserves terminal Beta Code elision as Greek koronis, with medial σ before the mark. An elided source token therefore cannot silently become a complete word; unelided homographs remain separate readings. After compatibility forms are restored, the exporter checks bare fragments again, including capitalized spellings. A terminal-consonant fragment with a matching case-preserving elided lookup reading is excluded unless a separate unelided owner or an independently edited headword establishes a complete word. Closed apocope and consonant-final word lists remain available. The export audit records these rejections; raw corpus counts do not override the check because transcriptions can lose apostrophes.

Exact reviewed corpus fragments are also excluded after compatibility and from source-alias recovery. The review for ἐνεί records the split complete words ἐνείραντα and ἐνείκασθαι and the remaining OCR loci; it preserves the independent headword ἔνει. The authored spelling review is reversible and leaves the original source readings unchanged.

The exact ἐπεῖ review distinguishes erroneous transcription from genuine ἔπει, ἐπεί/ἐπεὶ, and ἐκεῖ. The printed Herodotus witness has ἐπεὶ; the reviewed corpus contexts do not establish a circumflex verb form. The decision does not choose one replacement for unrelated damaged passages.

The optional data/ag_source_alias_audit.json and data/ag_source_alias_manifest.json preserve source readings and reversible alias changes, and pin the matching lookup, predecessor, source revision, and input hashes. scripts/build_source_alias_manifest.py verifies those pins and reverses each change before writing the manifest. Use --pre-alias-lookup for the first manifest, or --pre-alias-manifest to verify it against the recorded predecessor hash. When the audit includes historical closed-class elision checks, also pass the same --kaikki DIR to verify both external dump pins. The manifest verifies the imported classifier and helper against the source revision and retains strict byte reversal. These files are provenance artifacts and are not required at runtime.

Expansion sources sometimes encode restored letters with Leiden square brackets or spelling alternatives with parentheses. These delimiters are metadata, not Greek spelling. The build resolves only the structurally safe terminal movable-nu notation ((ν or (ν)) into both real spellings; all other bracketed forms or lemmas are rejected rather than guessed. The final overlay also cleans an already-expanded target, and build_lookup_db.py plus the Hunspell exporter repeat the rejection as defense in depth.

The GLAUx pair builder applies the same rejection before its shared tagged artifact is written. export_morphology.py repeats the check while deriving both movable-nu and elision tables, so grc_morph.json remains clean even when it is built from an older local pair file.

--expand-verbs does three things:

1. Classifies the headword via _classify_verb (suffix dispatch for thematic and athematic types, explicit table for εἰμί / εἶμι / οἶδα / χρή / φημί and their preverbed compounds) and runs Module:grc-conj on the present-system paradigm.
2. Parses the LSJ entry head paragraph with build/lsj_principal_parts.py to extract whatever principal parts LSJ has labelled (fut., aor., pf., pf. m./p., aor. p., impf., plpf.).
3. Re-invokes Module:grc-conj for each extracted tense, merging the results into the verb's paradigm. Verbs without parsable principal parts fall back to present-only behaviour, so the path is strictly additive.

About 712 LSJ-only verbs receive at least one parsed principal part through path 2; full-pipeline success on the 28,745 LSJ-only verb candidates is 99.27%.

A separate post-processing pass in build/build_grc_verb_paradigms.py takes the same parsed principal parts and procedurally synthesizes the missing finite-mood cells (subjunctive / optative / imperative / aorist infinitive) for thematic -ω verbs via build/synth_verb_moods.py, plus the full case×gender×number declension of every participle slot via build/synth_verb_participles.py, plus the past-indicative 1sg cells (active / middle imperfect; active / middle / passive aorist) for any verb where kaikki dropped the tense tag and the unaugmented Homeric variants got correctly filtered out into the epic dialect slice. Stem- templating only fills cells that aren't already attested by Wiktionary or GLAUx; real corpus cells are never overwritten. Corpus-only lemmas need an independent headword in data/grc_lexical_heads.json before this pass infers a citation or synthesizes forms. Without that evidence, the observed cells and dialects are retained, but no conjugation is extrapolated. Build the lexical-head inventory before building canonical verb paradigms.

Coverage: on the 27K-verb output the synthesis adds ~278K finite-mood cells, ~775K participle cells, ~15K aor-2 cells, ~430K contract cells (active + middle present participle), and ~36K past-indicative 1sg cells. Comparing against jtauber/greek-inflexion's hand-curated paradigms on the 3.6K shared lemmas, dilemma's per-mood coverage is currently:

mood	dilemma / jtauber	notes
indicative	0.83	mostly corpus / Wiktionary cells
subjunctive	1.17	full thematic + contract synth
optative	0.95	full thematic + contract synth
imperative	1.03	full thematic + contract synth
infinitive	1.49	over-broad sigmatic synthesis
participle	0.95	thematic + aor-2 + ε/α-contract
overall	0.97	well past the 95% full-flip mark

Synthesis covers thematic -ω verbs (regular and aor-2 / strong-aorist), α-/ε-/ο-contract verbs (present system), and the mixed-α aor-2 class (πίπτω → ἔπεσα, λέγω → εἶπα, εὑρίσκω → εὗρα) that uses α-style endings on the active and middle indicative but regular ο-thematic endings elsewhere. The participle ending tables encode jtauber's macron / breve quantity-mark conventions on feminine forms (-ουσᾰ / -ουσᾰν / -ούσᾱς present-active, -σᾱσα / -σᾱ́σᾱς aorist-active, -μενᾱς middle-passive, -υίᾱς perfect-active) so synthesised cells match jtauber verbatim.

Known limitations: athematic μι-verbs are handled in the upstream LSJ expansion (path 2) rather than the synth module; perfect contract participles aren't synthesised because they'd need separate perfect- stem extraction; ο-contract middle participles are skipped because jtauber's pattern is too inconsistent cell-by-cell; and a small set of verbs with idiosyncratic accent (e.g. ὁρᾷς) are deferred to corpus.

Non-Attic form filtering

GLAUx provides AGDT 9-position morph tags but no dialect axis: a Homeric unaugmented imperfect / aorist gets the same active-imperfect or aorist tag as its Attic counterpart, and a sandhi crasis form like κἄβλεψας gets the same active-aorist-2sg tag as a regular ἔβλεψας would. The build pass recovers dialect / sandhi status from the surface form itself before populating the canonical Attic slice:

• Crasis forms (consonant-initial words with a breathing mark on a non-initial vowel: κἄβλεψας, τοὔνομα, χἠμεῖς) are dropped entirely as textual artifacts. ~650 forms are removed across the 27K-verb output.
• Homeric iterative imperfects (-εσκον / -ασκον / -οσκον infix forms on verbs whose lemma doesn't natively end in -σκω) are routed to the epic dialect slice. ~370 forms.
• Athematic root-aorist passives (ἐλύμην / ἔλυντο / λύτο series, with middle-voice personal endings on a slot tagged passive aorist) go to epic. ~130 forms.
• Unaugmented past-indicatives (aorist / imperfect / pluperfect cells whose form lacks the augment that Attic mandates) go to epic. ~27K forms.

Detection is conservative: lemmas starting with ε- whose forms also start with ε- are skipped to avoid clobbering compound-prefix verbs where the augment is internal (ἐκμολεῖν → ἐξέμολεν), and lemmas with long-vowel initials (η-, ω-) are skipped because their temporal augment is morphologically invisible.

GLAUx does record a dialect per work, which build/build_glaux_pairs.py now carries onto the pairs: a form attested only in works marked Ionic, Epic, Doric or Aeolic is tagged with that dialect, so it fills that dialect's slice instead of competing for the Attic one. Works marked Attic, Attic/Koine or Koine share the default slice. Each pair also carries its token count (count) and, when its tokens were tagged more than one way, every analysis with its own count (analyses, [pos, tags, count], commonest first); pos and tags stay the first token's analysis.

Choosing the form that fills a cell

Source tags are interpreted deterministically. A single ambiguous grammatical axis contributes each explicit reading, and a form tagged for several dialects contributes to each slice. Tags with several ambiguous axes cannot establish which values belong together; they stay in the raw data but do not populate canonical cells. Filtering a collapsed cell does not discard another valid reading of the same form.

An unbreathed citation form can acquire a breathing from a unique surviving present first-person source cell when all other letters and marks agree. Existing breathings, accents, subscripts, and ambiguous candidates are preserved. This corrects the canonical citation without changing the original source rows.

For a verb whose own citation is explicitly marked defective, procedural synthesis is limited to the tense and voice families present in retained source cells, including dialect slices. An automatically inserted citation cell does not authorize a present paradigm, and a defective tag on a different referenced head is not inherited. Supplied cells remain unchanged.

Several sources can offer a form for the same cell, and the sources disagree. The pass ranks the candidates and, before ranking, discards two kinds of evidence that are wrong rather than merely weaker:

• A Wiktionary table row that reaches kaikki with its person and number collapsed onto the first cell. A real cell holds one form and at most a few spellings of it, so four or more distinct letter sequences in one cell of the same dialect means the row lost its person-number axis. Independently tagged Attic, Epic, Doric, and Koine variants are counted separately; rejecting one dialect's damaged cell preserves the others. The detector counts letters without diacritics; polytonic and simplified copies of contracted and uncontracted forms do not multiply that count. All surviving source variants remain available for choosing the canonical spelling. For example, τιμάω's entire present active indicative arrives tagged first-person singular, and τιμάς — far commoner as the accusative plural of the noun τιμή — then wins the cell on corpus frequency.
• In a subjunctive cell, a form whose ending belongs to the corresponding indicative. The subjunctive lengthens the thematic vowel, so -εις, -ομεν and -ονται rule a form out where -ῃς, -ωμεν and -ωνται are required. Endings the two moods genuinely share, such as the alpha contracts' -ᾶτε and the middle -ῃ, are left alone.

What survives is ranked by how many sources assign it to the cell, then (in a past indicative cell) whether it carries the augment, then whether Wiktionary lists it for this verb, then its corpus count in form_profile.db.

Finally, Wiktionary's conjugation tables decide the accent of any cell that a corpus token or the synthesizer spelled the same letters differently. That pass reads which cell Wiktionary puts each spelling in, so an accent that is the only thing telling two cells apart survives — λύσαι is λύω's aorist optative third-person singular and λῦσαι its aorist middle imperative — while a spelling both cells share is still corrected, as βαίνον is to βαῖνον. When grammatical tags exist, the replacement must fit the current cell, including an explicit tense. A missing tense imposes no extra restriction. An infinitive cannot reaccent a generated optative merely because the optative's own source row was filtered out. Untagged spellings cannot change a cell's accent, because they provide no grammatical evidence for that cell. Accent evidence also stays within the default paradigm's dialect: an Epic κυλίσαι does not replace Attic κυλῖσαι. A paradigm drawn entirely from another dialect uses that dialect's evidence. A unique matching cell takes precedence over the corpus's preferred accent. If a damaged table row survives in only some cells or dialects, only those retained readings can supply accent evidence.

Synthesized first-aorist middle imperatives in the second-person singular use recessive accent, with final -αι counted as short: κέλευσαι and ὄζησαι, compared with the active infinitives κελεῦσαι and ὀζῆσαι. Long penults written with η or ω, a diphthong, or explicit source quantity take a circumflex in two-syllable forms (κρῶξαι). The existing source-cell correction still supplies vowel quantity that an unmarked α, ι, or υ cannot establish. A diaeresis keeps adjacent vowels in separate syllables when placing the recessive accent (ἄϊξαι). Second-aorist mood synthesis requires a matching second-aorist principal part. An -α ending or a difference from the present stem is insufficient: sigmatic variants (ἔπαιξα) and liquid first aorists (ἔμεινα) must not generate second-aorist optatives or infinitives. Mixed-alpha synthesis is limited to established lemma/form pairs; other source forms remain in their attested cells. If the selected principal part does not fit the active second-aorist template, both mood and participle synthesis try the separately labeled second aorist. A middle-only principal part cannot displace an available active form. Passive aorist endings from a source table with an empty stem cannot supply a complete verb cell. For example, τύπτω's incomplete Ionic passive table lists ῆν and εῖτε; those readings are withheld while the complete active Ionic and passive Attic/Epic forms remain. This checks the specific grammatical cell and keeps independent homographs such as ἵημι's active aorist εἷτε and εἰμί's imperfect ἦν. The shared check also runs during Ancient source-table extraction, before creating lookup, POS, or training records. Raw filtering requires explicit, unambiguous grammatical axes. Gender-only participle headings are filtered only when the entry's matching grc-conj template explicitly lacks both passive stems and no complete competing template uses that dialect. Other missing or conflicting labels remain auditable source data. A separately identified active or middle reading survives when the same row's passive reading is invalid. For grc-conj aor-1, an augmented indicative stem without its nonpast stem can also produce bare active or middle endings. Extraction withholds only matching nonpast endings when every competing template for that voice and dialect proves the missing stem. Indicative-only declarations, complete competing templates, augmented indicative forms, uncertain labels, and independently supplied voice readings remain. This prevents incomplete οἰδάνω, δύπτω, and παρακελεύομαι tables from claiming forms such as ῶ and αίμην; it does not ban those spellings in other entries.

Periphrastic forms contribute the main verb, not the separate auxiliary εἰμί. The filter also recognizes the source's monotonic είμεν, είτε, and είεν copies when they explicitly carry perfect-optative tags. The distinct aorist forms of ἵημι and imperative forms of ἵστημι remain. Each marked auxiliary also needs its corresponding mood, preserving the aorist middle indicative ἔστω of ἵστημι beside the imperative of εἰμί. Conflicting mood labels, or conflicting tense labels on a monotonic copy, remain available for later disambiguation. Ancient Wiktionary extraction applies the same check before creating training pairs and lookup entries, so a compound tense cannot teach the isolated auxiliary as a form of the main verb. Original source dumps remain unchanged.

Ancient nominal headers can abbreviate a genitive ending, such as -ῆς beside αὐλή. An omitted hyphen does not make that ending a complete inflection. Unqualified header endings are withheld from training, lookup, and POS data; complete case forms, marked alternatives, and independent parts of speech remain available.

Export to ONNX

Generates ONNX model files so inference works without PyTorch.

python export_onnx.py                  # exports encoder.onnx + decoder_step.onnx

Shipping rebuilt data artifacts

build_lookup_db.py fills reviewed missing Modern Greek present-active cells for εμπίπτω, απορρέω, and συνιστώ, plus the masculine noun paradigm of αιτών. The rules in dilemma/el_paradigm_gaps.py require an existing self-mapped Modern head and never overwrite an existing Modern reading or infer another tense. The noun readings αιτούντος and αιτούντες belong to αιτών; they are not relabeled as finite verbs.

For a bounded refresh of an existing release, run python scripts/complete_el_paradigms.py --base PATH_TO_BASE --manifest PATH_TO_BASE_MANIFEST. The base directory must contain lookup.db, spell_index.db, and mg_lookup.json whose hashes match that release's manifest. The command uses the same rules as the full builder, preserves existing combined and Ancient rows, and writes data/changes/el_paradigm_completion.json with input/output hashes, source references, generator hashes, and every added row. Reversing the exact refresh with --undo PATH_TO_AUDIT restores the original logical rows; it refuses files changed by subsequent work. A full rebuild remains the standard path when other lexical inputs change.

The artifacts CI tests against (including lookup.db, spell_index.db, lemma_attestation.json, form_profile.db, and the ONNX/PyTorch files in model/combined-s3/) live on HuggingFace, not in git, so the two have to be kept in step by hand. data/hf_manifest.json pins their sha256 and scripts/hf_data.py does the bookkeeping:

python scripts/hf_data.py push      # upload whatever changed, refresh the manifest
git add data/hf_manifest.json       # commit it with the code change
./scripts/install_git_hooks.sh      # once per clone: pre-push sync check

CI waits for the Hub to serve the manifest's bytes before downloading (hf_data.py wait) and re-checks them afterwards (hf_data.py verify). That way a git push that beats a slow upload just makes the test job wait rather than run new tests against the previous artifacts, and an upload that never happened fails with a message saying so. The pre-push hook catches both cases before they reach CI; git push --no-verify skips it.

Release

Releases are cut with scripts/release.py. It requires a clean, synchronized main, updates VERSION, pyproject.toml, and CHANGELOG.md in lockstep, commits and pushes main, and waits for the Tests workflow on that exact SHA. Only after CI succeeds does it create and push the numeric annotated tag. The tag workflow verifies that successful main run, then publishes the GitHub Release and PyPI artifacts without running the full suite a second time.

python3 scripts/release.py 1.3.1 --dry-run
python3 scripts/release.py 1.3.1

Before bumping, regenerate and HF-upload any data outputs the new version is meant to ship (see the section above), so the release tag points at a SHA whose data files match what's on HuggingFace.

Testing

Tests run automatically on GitHub-hosted runners for pushes and pull requests to main. CI downloads data files from HuggingFace (lookup.db, spell_index.db, lemma_attestation.json, and the matching form_profile.db and form_citations.db pair for the Hunspell export, plus source tables including the elision-preserving diorisis_pairs.json) and the complete combined-s3 model, all pinned by sha256 in data/hf_manifest.json. ONNX Runtime and PyTorch are installed, and model/data integrity and real-model batched-decoding checks run before the full suite so missing or incompatible model files fail early. The Ancient export and coverage audit also run first, using the pinned merged corpus frequency table. The exporter creates its output directory before writing audit files, including on a clean checkout. The job has a 75-minute limit to cover both the full dictionary export and the test suite (see Shipping rebuilt data artifacts).

python -m pytest tests/ -v                  # run all tests via pytest (recommended)
python scripts/check_benchmark_regressions.py  # deterministic accuracy gate
python bench_fast.py                      # compact comparison across three periods
python tests/test_integrity.py              # data integrity + model inference checks
python tests/test_dilemma.py                # lookup table + end-to-end lemmatization tests
python tests/test_dilemma.py --lookup-only  # skip model tests

The compact benchmark counts missing predictions as errors and continues scoring the remaining forms.

The suite covers the runtime, source builders, exports, and their regression fixtures. tests/test_comprehensive.py is one of the largest single files (~170 tests) and covers core lemmatization, particle suffix stripping, verb morphology stripping, article-agreement disambiguation, crasis resolution, elision handling, orthographic normalization, dialect normalization (Ionic, Doric, Aeolic, Koine), convention switching, language filtering, spelling suggestions, batch operations, and edge cases. The remaining files cover paradigm builders, athematic verb expansion, LSJ principal-parts extraction, the morph_diff annotator, and end-to-end integrity checks.

tests/test_integrity.py runs 7 structural checks: ONNX/vocab dimension match, DB table presence, model load, inference, and ONNX/PyTorch parity. tests/test_dilemma.py validates lookup correctness and known form-lemma pairs across Greek varieties.

Frequency-ranked inflection lists

rank_forms.py produces per-lemma ranked form lists, sorted by corpus frequency. This is useful for downstream consumers (e.g. Lemma) that need to know which inflections of a word are most common.

python rank_forms.py --lang el     # Modern Greek (default)
python rank_forms.py --lang grc    # Ancient Greek
python rank_forms.py --lang mgr    # Medieval/Byzantine Greek
python rank_forms.py --lang all    # All three

For each language, the script produces:

• {prefix}_ranked_forms.json - lemma to list of forms sorted by frequency
• {prefix}_form_freq.json - form to raw frequency count

Frequency sources (used for primary ranking):

• MG: FrequencyWords/OpenSubtitles (1.49M forms)
• AG: merged corpus_freq.json (94.0M tokens, 1.99M unique forms) from GLAUx + Diorisis + PatristicTextArchive + the Open Greek Corpus open-text rollup (First1KGreek, corrected Patrologia Graeca, Perseus canonical-greekLit, byzantium.gr)
• Medieval: AG corpus frequencies as proxy

Additional corpora available in --verbose per-form breakdowns:

• Patrologia Graeca (freq_pg): 3.14M tokens of Byzantine/Patristic Greek from PG corpus (Church Fathers, PG071-PG158)
• Byzantine vernacular (freq_byz_vern): 191K tokens from the Byzantine Vernacular Corpus (Digenes Akritas, Chronicle of Moreas, Erotokritos, etc.)

By default, rank_forms.py downloads pre-built files from the open-greek/dilemma-data HuggingFace dataset. Use --rebuild to regenerate locally from the lookup and frequency source files:

python rank_forms.py --lang el --rebuild            # regenerate MG locally
python rank_forms.py --lang all --verbose --rebuild  # all languages with per-corpus breakdown
python rank_forms.py --lang el --polytonic --rebuild  # MG with polytonic variant ranking

The --polytonic flag generates mg_polytonic_ranked.json, which maps each monotonic MG form to its attested polytonic variants ranked by corpus frequency. This is built from mg_polytonic_freq.json (see build/build_polytonic_freq.py), which extracts polytonic word frequencies from the glossAPI/Wikisource Greek texts dataset (~38M tokens, 5,394 texts). Forms appearing fewer than 3 times are filtered out.

The --verbose flag adds a {prefix}_ranked_forms_verbose.json file with per-corpus frequency breakdowns for each form. Each entry includes frequencies from all available corpora (OpenSubtitles, GLAUx, Diorisis, Patrologia Graeca, Byzantine vernacular), letting consumers re-rank for mixed-period use cases (e.g., a Modern Greek dictionary for a book about ancient topics could boost forms with high freq_glaux).

Hunspell spell-check export

export_hunspell.py produces compact Hunspell .dic + .aff pairs from lookup.db, aimed at mobile consumers (primarily the Tonos iOS polytonic keyboard) where the full 1.07 GB lookup.db and 560 MB spell_index.db do not fit inside the ~48 MB memory ceiling of a keyboard extension. Affix compression uses a zero-strip SFX rule only when its dictionary base is itself a real member of the paradigm. A mechanical common prefix such as λόγ or ἀνθρώπ is inlined instead, because both Hunspell and Tonos accept every .dic base as a word. Every flagged base therefore has an identity rule. Editorial brackets and parentheses are excluded before affix compression, so the generated .aff remains loadable by consumers that compile affixes as regular expressions, including spylls.

Default output is the grc variant (Ancient + Medieval polytonic), which is what Tonos ships. An optional el variant (Modern Greek monotonic) is retained for other downstream consumers via --variant el. Output layout under build/hunspell/:

Variant	Script name	Lang tag	Contents
grc_polytonic.{dic,aff,version}	grc	grc	Ancient + Medieval polytonic forms. Rows owned by src='grc' are eligible, as are language-shared rows for a lemma that also has grc evidence; this preserves AG headwords such as λέγω, πατήρ, and γῆ without admitting unrelated Modern lemmas. New forms require exact evidence from the complete accent-preserving form_profile.db, including hapaxes, or membership in a pinned citation, textbook-paradigm, grammar, or productive second-accent class; arbitrary unattested generator output is not admitted. A productive second accent needs a proparoxytone or properispomenon host (θάλασσάν, δῶρόν, but not λύκοί). A new spelling that differs only in accent, breathing, diaeresis, or iota subscript from a well-formed spelling with at least 1,000 corpus tokens (ἑγώ beside ἐγώ, ταΐς beside ταῖς) must also be attested in the GLAUx or Diorisis treebanks: at least 25 tokens when it has under 1% of the common spelling's count, at least 5 when it has under 5%. A spelling both treebanks annotate independently, more than once each and at a rate of at least one treebank token per thousand of the dominant spelling, is confirmed instead of counted, which keeps the dual πρώτω, Doric γλώσσᾳ and the contract πειρᾷς. This keeps genuine dialect spellings such as Doric τᾷ and τῶ. Outside the reviewed April surface (see the release audit below), a vowel- or rho-initial form without a breathing is rejected, and so are a grave accent before the final syllable (καὶτοὺς, ὓστερον) or on an elided word (γὰρ᾽), and a second accent anywhere but on the ultima of a proparoxytone or properispomenon, or before a fused enclitic (Αἴγυπτόνδε). Valid elisions normalize their final textual apostrophe to U+1FBD koronis. Same-lemma bare elision fallbacks are excluded unless an independent DGE/Cunliffe headword proves the collision, while the reviewed historical fallback set catches malformed self-headwords and preserves genuine ἄν. The closed Homeric apocope set preserves κὰτ/κάτ through κὰγ/κάγ and ἂμ/ἄμ. A closed grammatical list preserves legitimate unaccented words: enclitics including the τις, φημί, and εἰμί forms, dialect proclitics (ἁ, αἰ), and crasis with a proclitic (κἀν, χὠ), including the legitimate indefinite-pronoun forms του and τῳ. The same list carries the article's graves (τὸ, τὼ), which lookup.db leaves to resolve_articles, the relative pronoun's (ὃ, ἣ, οἳ, αἳ), and the crasis of καί written with an iota subscript that GLAUx or Diorisis has at least 4 times (κᾂν, κᾀπὶ, κᾀπειδὰν, κᾄπειτ᾽, the unaccented κᾀκ and κᾀξ), for which lookup.db has no row. A spelling GLAUx or Diorisis has at least 5 times outside the language model's held-out sentences comes in even when no lemma source proposes it, under itself, or when only the Modern Greek lexicon does, under that lexicon's lemma (σπανίως, κτήσεις, χαίρετε); the counts are case-sensitive, a contextual grave counts with its acute, and the held-out sentences' own occurrences are left out (data/hunspell_grc_lm_dev_spellings.json.gz, written from build/lm/dev_sentences_{glaux,diorisis}.txt by scripts/build_hunspell_dev_spellings_fixture.py), so those sentences stay a fair test of the dictionary. An elided spelling, an inner capital, an iota adscript, a vowel-length mark, a breathing after a consonant (δὔ, a converted elision mark), a second accent, or the grave of an explicitly rejected spelling (τοὺ) keeps a treebank spelling out, a capitalized one comes in only when its lowercase is not a word, and the respelling rule above applies to them as to every new form; --treebank-spelling-min-count 0 turns both rules off. Crasis inside a word may carry its breathing past the first syllable (ἐγᾦμαι, καλοκἀγαθία, ταὧν), crasis with ὦ keeps the interjection's circumflex wherever it lands (ὦνθρωπε, ὦλεθρε), and reviewed lists keep complete words that end in a consonant (ἔκ, παρέκ, ὑπέκ, χερουβίμ). A fused enclitic's host keeps its own accent window: the -τις of ὅστις (οὗτινος), -περ (οἷονπερ), and the -δε of the epic and Ionic datives of ὅδε (τῇσιδε); any other -δε takes the second accent (πόλεμόνδε). In a Latin name ου between two vowels is the consonant v, not a syllable (Ὀκτάουιος), and the Ionic enclitics τεο, τεῳ and τεων are written without an accent. Every retained contextual grave receives its citation/before-pause acute twin unless the twin fails a structural rule.
el_GR_monotonic.{dic,aff,version}	el	el_GR	Modern Greek monotonic forms, including MG-relevant vocabulary drawn from the AG side of lookup.db (articles, common verbs, proper names). Not shipped in Tonos.

A third list, grc_mg_polytonic.{dic,aff,version}, holds polytonic Modern Greek spellings, attested ones and generated ones for the frequent forms of verbs, nouns, adjectives and the other inflected words, for a keyboard to merge into grc. It comes from export_mg_polytonic.py; see below.

Each dictionary entry carries a morphological field fr:<bucket> where the bucket is one of C (common), M (medium), R (rare), or X (unseen in corpus). This lets consumers rank spelling candidates without shipping the full frequency table.

The bucket is chosen from three signals, in priority order:

1. Canonical seed. data/canonical_ag_forms.json pins ~194 iconic polytonic surface forms to C (Iliad/Odyssey/Herodotus incipits, Olympians, Homeric heroes). These are low-token-count forms whose cultural weight exceeds their corpus frequency - ἄειδε appears only 71 times in corpus but is the opening word of the Iliad. The grc export also pins the citation forms of the 2,000 most frequent LSJ9 headwords and the 971 reviewed forms of the language-model head, so unaccented λυω ranks λύω first. Textbook paradigm cells and closed-list forms are exempt from the evidence filters but keep their corpus bucket.
2. Canonical lemmas. ~168 famous lemmas (ἀείδω, μῆνις, Πλάτων, etc.) promote any polytonic-marked form of theirs to C. This catches canonical inflections beyond what the seed enumerates.
3. Lemma aggregate. For polytonic forms, the lemma's total corpus count promotes the form: >= 20K -> C, >= 5K -> M. Highly- inflected lemmas (ἀείδω has 1289 surface forms) would otherwise dilute frequency across the paradigm so no single form crosses the per-form threshold.
4. Per-form count fallback. count >= 1000 -> C, >= 100 -> M, >= 1 -> R, else X.

Only polytonic-marked forms receive the canonical promotions, so monotonic leaks like άειδε (acute-only, no breathing) stay at R and downstream rankers can pick the polytonic variant when the two otherwise tie.

python export_hunspell.py                 # grc polytonic (default)
python export_hunspell.py --variant both  # grc + el
python export_hunspell.py --variant el    # el monotonic only
python export_hunspell.py --sanity 10000  # 10K-lemma sanity pass (no April surface)

The release audit loads the expanded .dic/.aff pair with spylls and checks both the corpus head and the complete artifact. The LM fixture lists the 1,000 most frequent Greek-bearing words of the language model (ranked with a contextual grave read as the acute) in each of the 1,215 spellings the model writes them with, καί and καὶ apart (16,546,390 LM tokens). Each spelling is looked up as written: a keyboard writes a grave only where the dictionary has the grave spelling, so an acute twin does not stand in for it. All must be accepted except 30 reviewed Modern spellings (among them the graves τὴ and στὴν, which the polytonic Modern Greek list carries), fragments, numerals, bare stems and corpus misspellings (ὃτι, ἢδη), which must remain rejected. The fixture checks all 1,185 remaining spellings, including their contextual grave variants. The whole-artifact gates additionally check the 1,363,275 spellings in the reviewed baseline (the export of commit 2e633cc, as the keyboard compiled it), with documented exclusions for unsupported case variants, bare elision fragments, reviewed respellings, and explicit source errors. The current export has 1,371,104 expanded spellings. The gates also require all 1,996 valid citation spellings among the top 2,000 LSJ9 lemmas, and all 2,154 pinned cells of the textbook paradigms for λύω, παιδεύω, τίθημι, δίδωμι, ἵστημι, τιμάω, ποιέω, and δηλόω. They also reject punctuation, internal breathings, impossible accent placement, synthetic flagged bases, bare elision stems, and new weak respellings of common spellings, and require an acute twin for every contextual grave whose acute spelling passes the structural rules.

The baseline is the dictionary Tonos ships now, not the April 0.4.1 export it started from: it moves forward with each swap-in, and what it encodes is that an export must not lose what the keyboard already accepts. Because the gate strips its structural classes before pinning, a baseline taken from a recent export grandfathers far less than one taken from April: 18 truncated stems rather than 8,971, and no missing-breathing forms rather than 166,226. The reviewed surface is exempt from Dilemma's structural rules, and whatever still fails them stays until it is reviewed in Tonos, because dropping it would count as a loss in the gate. The acute twin of a reviewed grave inherits its review.

data/hunspell_grc_spelling_review.json records spellings a review rejected, which the exporter drops with their grave or acute twins whatever protects them, and which the audit no longer requires. Two reviewers labelled each candidate independently, from its attestation by source and up to three corpus sentences, and a form is listed only where both rejected it. It covers two groups: respellings of common words that OCR'd text admitted (191 of them, such as στρατηγού for στρατηγοῦ and βοής for βοῆς), since the respelling guard's thresholds were calibrated on labelled forms and are not to be tightened by guess; and reviewed-baseline entries that fail the structural rules and are not words anyone types (89: editorial underdots, synizesis ties, spurious extra accents such as δόκιμάζω). Two breathing variants both reviewers rejected, αὗθις and ἄπαντα, stay, because GLAUx attests them in held-out text. The exporter also accepts the polytonic Modern Greek particles θὰ and γιὰ, which no lemma source proposes.

The reviewed source specifications also preserve παῖσι as the old Attic dative plural explicitly recorded under παῖς in LSJ. Its dictionary evidence is kept separate from mistaken accents in later corpus transcriptions.

A finite reviewed source-form list can preserve a rare spelling beside its more common form. For example, ὤοντο remains beside ᾤοντο only while its exact spelling is present in both cited Diorisis passages and the merged lexical inventory supports οἴομαι. The audit distinguishes the reviewed grammatical interpretation from the corpus's unknown lemma. The export requires matching logical content hashes and source hashes in form_profile.db and form_citations.db before joining their form IDs; rebuild both with build/build_form_attestation.py when either changes.

A content-hashed fixture measures rejection counts over the New Testament (Perseus), the Septuagint (First1KGreek), Iliad 1, Herodotus 1, and a Katharevousa sample. These are regression corpora, not held-out text: form_profile.db contains the same works, so the exact-attestation rule admits almost all of their forms. Candidate token and type rejection counts, measured with spylls, may not exceed those of the April .dic/.aff read the same way. data/hunspell_grc_shipped_compat.json.gz records the reviewed baseline surface; data/hunspell_grc_textbook.json.gz pins the textbook paradigms, taken from data/ag_verb_paradigms.json (built by build/build_grc_verb_paradigms.py; the fixture records its SHA-256); tests/fixtures/hunspell_heldout.json.gz records only exact form counts and source hashes, not source prose. The compatibility fixture is built from files that ship with Tonos, not with Dilemma: its compiled dictionary and language model, and its candidate gate with the gate's review lists, each recorded by SHA-256. Rebuild the fixtures only when deliberately changing the baseline or source revisions:

python scripts/build_hunspell_compatibility_fixture.py \
  --baseline /path/to/shipped/grc_polytonic \
  --lm /path/to/shipped/grc_ngram.bin \
  --gate /path/to/candidate_gate.py
python scripts/build_hunspell_textbook_fixture.py
python scripts/build_hunspell_heldout_fixture.py \
  --baseline /path/to/shipped/grc_polytonic \
  --ogc-root /path/to/open-greek-corpus

The corpus-head fixture is built from the full train_lm.py run behind the current language model (--source json: vocab.json, unigrams.json and stats.json, 30,933,396 training tokens), not a train_lm.py --sanity output. JSON fixture regeneration reads stats.json and fails unless "sanity": false; the binary route reads the exact vocabulary and unigram counts embedded by export_lm.py. The exporter itself uses data/form_profile.db (python -m dilemma download --with-attestation) for sparse exact-form evidence; data/hunspell_grc_form_freq.json.gz is retained only as an auditable LM artifact and is not an admission boundary.

Sparse inflections can use an exact dictionary POS reading when a flat lookup self-map hides their defined headword. data/grc_contracted_verb_aliases.json records explicit contracted-verb references separately from independent definitions, so an attested form under καταμετρῶ can retain the defined καταμετρέω owner. The exporter requires one supported target and preserves tied references without choosing between them; it does not infer a verb from its ending. Contracted references retain the source spelling's accent; only contextual grave and acute variants share a key. An accent-tolerant lookup alias cannot borrow a contracted citation's evidence when the spelling itself differs from that citation only in its marks. Build this artifact with python build/build_grc_contracted_aliases.py --kaikki DIR after generating the lexical inventory from those same source extracts. The export excludes the monotonic lookup aliases of the genitives and datives of λιγυστικόν and περσικόν; their circumflex spellings and the acute dual περσικώ remain available.

# Audit a fresh Hunspell export against the committed full-run fixture.
python scripts/audit_hunspell_frequency.py --source fixture

# Regenerate after a full train_lm.py run; refuses sanity outputs.
python scripts/audit_hunspell_frequency.py --source json --write-fixture

# Or regenerate from a format-v2 binary and its provenance sidecar.
python scripts/audit_hunspell_frequency.py --source binary \
  --lm-binary build/lm/grc_ngram.bin \
  --lm-version build/lm/grc_ngram.version \
  --write-fixture \
  --attestation-out data/hunspell_grc_form_freq.json.gz

# Produce the full LM-weighted review against a previously shipped artifact.
python scripts/audit_hunspell_frequency.py --source binary \
  --lm-binary build/lm/grc_ngram.bin \
  --lm-version build/lm/grc_ngram.version \
  --baseline-dictionary /path/to/shipped/grc_polytonic \
  --comparison-report build/hunspell/baseline_comparison.json

Output layout, with one sidecar .version file per variant so the consumer can detect updates:

build/hunspell/
  el_GR_monotonic.dic
  el_GR_monotonic.aff
  el_GR_monotonic.version   # semver + dilemma commit hash + entry count
  grc_polytonic.dic
  grc_polytonic.aff
  grc_polytonic.version
  grc_mg_polytonic.dic      # from export_mg_polytonic.py
  grc_mg_polytonic.aff
  grc_mg_polytonic.version
  eval_results.txt          # from eval_hunspell.py

eval_hunspell.py is the quality gate. It samples mid-frequency real Greek words (ranks 100..10K in the corpus), generates synthetic typos at edit distance 1 and 2, and measures top-1/top-5 correction accuracy against the compact artifact. Add --compare-full to also benchmark Dilemma's own suggest_spelling() on the full lookup.db.

python eval_hunspell.py                    # default 500 targets per variant
python eval_hunspell.py --n 100            # quick sanity eval
python eval_hunspell.py --variant el       # only MG
python eval_hunspell.py --compare-full     # compare vs full Dilemma (slower)

Requires pip install spylls for the Python-side Hunspell consumer. To regenerate both artifacts and the eval report end-to-end:

python export_hunspell.py
python eval_hunspell.py --n 200

Polytonic Modern Greek word list (grc_mg_polytonic.dic)

The grc dictionary is built from Ancient and Medieval Greek sources, so it has almost no Modern Greek vocabulary, and on purpose it rejects the unaccented του and the Modern spellings among the language model's commonest words (κι, της, μας, τώρα). A keyboard that ships only grc turns polytonic Modern Greek (Modern Greek written with breathings and accents, as in print before 1982, in church texts and by some writers today) into Ancient Greek neighbors: τώρα becomes τἄρα, ἀκόμη becomes ἀκμὴ, σπίτι becomes σπίτιν. export_mg_polytonic.py writes a separate list that such a keyboard can merge into grc, and leaves grc_polytonic.dic as it is:

python export_mg_polytonic.py     # build/hunspell/grc_mg_polytonic.{dic,aff,version}
python export_mg_polytonic.py --holdout-author-fold 0/5 --out-dir DIR   # evaluation list

The .dic is a plain list, one spelling per line with the grc fr: field and no affix flags; the .aff only makes the pair loadable on its own. No entry is a spelling grc_polytonic.dic accepts, as written or, for a capitalized spelling, through its lowercase entry, so the whole list is Modern-only. The .version sidecar records the sha256 of the grc dictionary the list was selected against, and the exporter must be rerun after every grc rebuild. An evaluation variant needs --out-dir and is refused in build/hunspell, where it would replace the shipping list. The source is the polytonic Modern Greek slice of Wikisource that the next-word language model also reads (extract_polytonic_mg.py), and only the language model's training sentences, so its dev sentences stay held out; a training sentence that repeats a dev sentence word for word, case aside, as Wikisource's duplicate copies of a text do, is left out too. A spelling is listed when:

• it comes from a polytonic source. A document is dropped when more than 1% of its lowercase words, with 95% confidence (the Wilson lower bound), and at least three, read as monotonic: a vowel-initial word without a breathing, or a monosyllable polytonic writing always accents (και, να, δεν, στο, ...), counted on the training sentences. The bound keeps a short polytonic poem with three OCR slips; it drops 39 of 2,213 documents, 10.5% of the slice's words. Inside the rest, a sentence with any such word is skipped, which also drops the monotonic license note Wikisource adds to its pages. Elision and aphaeresis marks are read as the texts write them: the apostrophes, the koronis, and the spacing psili and dasia (νά ῾βρω), which in front of a vowel is that vowel's breathing (῾Η is Ἡ);
• it has at least 3 tokens from at least 2 authors, in its own case: the lowercase spelling counts its lowercase tokens, a capitalized one (a name) its capitals inside a sentence, and a capital that opens a sentence, line, quotation or stretch of dialogue, or follows an ellipsis or a closing bracket ([Γελῶσα] Χάχ), goes with the case in which more authors write the word inside a sentence, the word's grave and acute spellings together. A capital carrying the iota (ᾍδη) is a capital, though Unicode makes it a titlecase letter. So a word and a name are both listed when each qualifies (διάολος), and neither passes on the other's authors (λάζος, one author's "blade", beside the name Λάζος); a capitalized spelling whose lowercase is listed adds nothing a Hunspell reader does not accept, and is left out. Measured with each dev word's author held out (below), one-author spellings would add 3,200 entries that cover 0.8% more of the dev words when their authors are read and 0.2% more when they are not; a floor of 2 tokens would add 9,100 entries, of which 60% have a monotonic spelling Dilemma's lexicon knows, against 68% at 3 or 4 tokens;
• it is well-formed polytonic Modern Greek: the grc structural rules and explicit rejects, in either case, except that the weak possessive pronouns του, της, μας, σας, τους, των and κι are correct unaccented, syllables are counted with synizesis (τέλειωσε, πλάγιασε, γάϊδαρος keep their accent within three), a lowercase word may end in a consonant when it is not an attested word that lost its last vowel (ἄχ, κονιάκ, but not τόμ), and an elided word may end in the glide ι (κι᾽, γι᾽). An unaccented elided spelling must stand for an attested word accented on the vowel it lost (γι᾽, γιατ᾽, ἐδ᾽), so μητ᾽, φερ᾽ and μεσ᾽ are out: a paroxytone keeps its accent (μήτ᾽, μέσ᾽). The weak accusatives το, τα, τον, την stay out: a bare-typed το is almost always the article τὸ;
• it is not a misspelling of another spelling in the slice. A spelling whose breathing alone differs from another is one when the other is five times as common, or at least as common and accepted by grc, however rare both are (ἔτοιμος beside ἕτοιμος, εἷνε beside εἶνε); a reviewed list keeps the real homographs (ὅντας, ἄρματα, οὗλα). Any other mark-only respelling with under 5% of the count of a spelling of the same letters that has at least 100 tokens is one too (εἴχε beside εἶχε), unless its monotonic spelling differs and Dilemma's lexicon knows it, which keeps different words (χρονιά beside χρόνια, ἔμενα beside ἐμένα); such a word must still not be that rare beside the commonest spelling with its own monotonic letters and accent (ὄποιος goes beside ὅποιος). An elided spelling counts as another word only through an attested word it stands for (κάθ᾽, for κάθε, beside καθ᾽);
• a recorded review did not reject it: data/hunspell_grc_spelling_review.json and data/hunspell_mg_spelling_review.json, which lists the accent-type and iota-subscript misspellings no rule catches (ἤταν, ἐδῷ, ξῦλον, χειμώνα beside the slice's five times commoner χειμῶνα), two words run together (τουκαὶ, and εἰδ᾽, written for εἰ δ᾽) and τουναντίον without its koronis.

Each oxytone also gets its contextual twin (στὴν beside grc's στήν). fr: is the higher of two buckets: the grc bucket edges on the spelling's token count in the slice, the two twins counted together, and the bucket of its monotonic form's count in monotonic Modern Greek text (data/mg_freq.txt, OpenSubtitles, 263M tokens) at the rates per token of grc's edges on its 27M-token corpus: C from 37 per million (9,724 tokens), M from 3.7 (973). A traditional subjunctive in -ῃ or -ῃς counts with its monotonic spelling in -ει or -εις, and an elided spelling has no monotonic count. The counts are of lowercased text, so they count a name with the word of the same letters: a lowercase spelling whose monotonic form the slice writes more often capitalized inside a sentence, with 95% confidence (the Wilson lower bound of the capitalized share above one half), gets no bucket from them (κίνα, beside Κίνα), nor a capitalized one the slice does not write so. A handful of capitals is no such evidence: the slice's 4 Τσάντα beside 2 τσάντα say nothing of the noun's 18,966 monotonic tokens. A name the slice does not write still lends its tokens to a generated form (the imperative κίνα is M, and μάσα); Wiktionary's list of names is no guard, since it lists rare ones too (Σταμάτα, beside the imperative σταμάτα). A generated form ranks no higher than an attested spelling of its letters, whatever their counts: the imperative μίλα, C by its own, is M like the attested μιλᾶ. A spelling with the second accent an enclitic gives it (ὁδήγησέ) is the same word as the plain one and sets no such ceiling. The slice is 1.7M tokens of older prose, so its counts alone put 234 spellings above R; monotonic text raises 3,109 more from R (βρήκαμε, ἀγόρια, καλοκαῖρι, λέμε) and 120 from M to C, and gives the generated forms below their only bucket. A spelling in -ῃ or -ῃς takes only its share of its -ει form's count (below). The .version records where the buckets come from (bucket_source). A keyboard may take a C or M word for a sign that a text is Modern Greek, or write letters that only such a word has as that word, so these get fr:R whatever their count:

• one the GLAUx and Diorisis treebanks attest 20 times (του, the Ancient enclitic genitive of τις), or the contextual twin of a grc spelling they attest that often. A treebank count is the larger of the two treebanks' counts: they annotate largely the same works, so their sum counts most tokens twice (τὴ: 15 and 11, 9 of them the same lines of Aristophanes), and taking each work from one of them would lose a word the other spells differently (οὔτε: GLAUx 0, Diorisis 13,323);
• one the patristic texts (PTA) or First1KGreek write, with its twin, 20 times and at a rate at least a thirtieth of its rate in the Modern Greek slice (μοναχὸς, beside the patristic and Byzantine μοναχός). Below that rate a Modern spelling in those OCR'd editions is a misprint or a quotation (στήν, δὰ), so στὴν and στὸν stay signs of Modern Greek;
• one whose letters grc has under no spelling while the treebanks write them 20 times: the interjection τάκ, whose letters are those of the crasis τἀκ, would otherwise turn a passage of Ancient Greek that writes τἀκ into Modern Greek.

The Patrologia's OCR is not read, nor the Byzantine vernacular corpus, which is early Modern Greek itself. The counts come from Dilemma's form attestation (data/form_profile.db, python -m dilemma download --with-attestation); without it the exporter warns and caps nothing. In the current list this caps του, των, μοναχὸς, μίας, κό, πὲ and τάκ, with their twins. The .version records the sha256 of every file that decides the list: the grc dictionary, lookup.db, form_profile.db, and for the generated forms and the monotonic buckets the verb paradigms (data/mg_pairs.json) and the monotonic counts (data/mg_freq.txt).

The list also names the grc spellings polytonic Modern Greek writes another way, with lines of the form form<TAB>mg:avoid, so that a keyboard writing Modern Greek can leave them out of its candidates: the Katharevousa dative ἐκείνῃ beside ἐκείνη, misspellings such as εἷναι beside εἶναι or μὰς beside μᾶς, and the weak forms με, which the slice writes μὲ 25 times as often, σε, που beside ποὺ and ποῦ, and πως. A lowercase spelling is marked when it is a respelling of its word, the spellings of the same letters with the same monotonic spelling (accents as the acute, breathings and iota subscript dropped):

• its share of the word's lowercase tokens is under a tenth with 95% confidence (the Wilson upper bound, which needs at least 53 tokens of the word), and the word's commonest spelling takes at least a tenth, is a word grc or the list accepts and is written by at least as many authors. Another word of the same letters does not count: ὁπού, 26% of the demotic relative's spellings, is not marked though ὅπου fills its letters;
• a spelling of the word that the slice writes more often and grc or the list accepts differs from it only in marks, and where they differ in the breathing, the accent type or a dropped iota subscript, the GLAUx and Diorisis treebanks do not write the rarer one 20 times and a twentieth as often as the other, each with its contextual twin and both treebanks counted. An added subscript, with the accent it brings, is a respelling outright: polytonic Modern Greek writes the Katharevousa dative ψυχῇ as ψυχή, while the treebanks, where the dative is a word, cannot say so (taking the datives to the treebanks too unmarks 45 lines and costs 15 words on the folds below). The test keeps the Katharevousa reflexive αὑτοῦ beside αὐτοῦ, ὄν beside ὅν, ἤν ("if") beside ἦν and the relative ἥ beside the dative ᾗ. The breathing misspellings the slice writes (εἷναι, ἃν, ἄμα, ὦν) are at 3.3% of their word or below, but a misspelling whose spelling is also an Ancient word goes unmarked (ὀποῦ, the genitive of ὀπός "sap", beside the relative ὁποῦ; σάς beside σᾶς), and the rarest reflexives (αὑτά, αὑτήν, αὑτοῖς, 4.0 to 4.6%) stay marked, as does the subjunctive ᾖ, which adds a subscript to ἤ.

So a spelling that is another word is never marked: ὅ of ὅ,τι, the numerals ἕν, εἷς and ἕξ, πάρα, ὥς, πασᾶν and the crasis ταὐτά differ from their commoner neighbors in the monotonic spelling. The weak forms με, σε, που and πως are reviewed exceptions: unaccented, they are words of their own (clitics and Ancient enclitics), but polytonic Modern Greek writes them accented in most of their uses. So is τῶ, the τῶν of Εφταλιώτης and Μαβίλης without its final ν, which has the monotonic spelling of the dative τῷ and is never marked. Nor is a word with the second acute an enclitic gives it (γυναῖκά beside γυναῖκα), a spelling the list or its contextual twin carries, or one of the unaccented closed list. Elided and aphaeresized spellings are not judged, and only grc's spellings get a line, with the contextual grave of an avoided acute. A Hunspell reader that does not know the field accepts the marked spellings, as grc does.

The list has 16,774 attested words (759 in C and 2,408 in M), 49,344 generated spellings (below), 189 mg:avoid lines and 6,651 mg:shared lines. The attested words include 969 capitalized, 152 elided and 197 aphaeresized spellings (᾽ναι, ᾽ς), the six weak pronouns, κι and κι᾽, στὴν, στὸν, στὴ, στοὺς, τὴ, γι᾽, and the Modern vocabulary (τώρα, ἀκόμη, σπίτι, ὄχι, μπορεῖ). θὰ, νὰ, γιὰ, στὸ, στὰ, δὲν, ἀπ᾽, μου and σου are in grc already. An adversarial review of the earlier 17,789-entry list judged 500 random entries and every entry whose breathing, accent-type or iota-subscript twin is five times commoner, and confirmed 82 errors, about 1 in 200; the rules and the review file above remove all of them but ποιητῆ, which a second look found correct (the genitive, beside the accusative ποιητή). Of a new random sample of 200 entries none is an Ancient-only or OCR-damaged spelling; the rarest are older spellings and dialect words (θᾶμμα, κἀμμία, ὁλοκαιρὶς, μίλαε, crasis such as σὅλους). A later review found no error in 300 random entries (every C, 100 M, 179 R) and two in 50 each of the capitalized, elided and aphaeresized ones: εἰδ᾽ (now in the review file) and Χὰχ, an interjection that only sentence-initial capitals had made a name; a scan of every capitalized entry added Χάχ, Ἀετὲ, and ᾽Σ beside ᾽ς. The rules above now leave all of them out.

The slice is mostly older prose, so it rarely attests the second person and the spoken forms of common verbs, or the words of everyday life: it has ἔρθει and ἔρθουν but not ἔρθεις, no μιλήσεις, πιεῖς or καταλάβεις, and no τηλέφωνο, κινητό, αὐτοκίνητο or ἐνημέρωση. A keyboard with the attested list alone turns these into Ancient neighbors (μιμήσεις, πιεῖν, the datives αὐτοκινήτῳ and κέντρῳ), and with the verbs generated but not the nouns, into verb forms of the same letters (τηλεφωνῶ, ἐνημερώσῃ). mg_polytonic_paradigms.py writes polytonic spellings for the inflected forms of every part of speech in Wiktionary's monotonic paradigms (data/mg_pairs.json, written by build_data.py): verbs, nouns, adjectives, names, adverbs, pronouns, numerals and the rest, but not romanizations, abbreviations or symbols. It does so for every form that monotonic text has at least 50 times (data/mg_freq.txt, OpenSubtitles). A cell Wiktionary tags rare, archaic, dialectal, obsolete or dated is generated like any other (ὑπακούουν, συντριβεῖ, ἀπεχθάνεσαι), unless another part of speech has its letters, whose count it would borrow (πιάνου, the piano's); Katharevousa cells and datives are not generated. These Wiktionary rows are left out:

• forms filed under an abbreviation in capitals (ΔΕΣ for δες), whereas a page title in a heading's capitals is read in lowercase (Ερωτώ, with ρώτησα);
• the pieces of tables Wiktionary could not split into stem and ending (σαλπάρω's ούμε, όμαστε, όσουν). Such a piece starts with another letter than its word, even after an augment, and is the very ending of the same cell in at least 5% of the words of its part of speech; a suppletive form (φάγατε, είπα, ήρθα) also starts otherwise but ends no other verbs' forms;
• spellings that are polytonic already (ἔχομεν, ζᾶ);
• a noun's or adjective's row with no case or number where another row of its part of speech places the form (αρκτικών, filed only as a form under the sentence-case page Αρκτική, beside its cell under αρκτικός);
• a name's form with the letters of another word (the surnames Δώση, Γράψη), which is that word at the start of a sentence, and the names Wiktionary does not decline in any form (Τέσα, Τζέιμς, Γουίλιαμς), foreign names in transliteration, which Greek writes in many ways.

A form several words share gets one spelling, from the word that is a verb rather than another word's page, then the one whose letters it shares the longest beginning with (πουλᾶμε is πουλάω's, not πωλώ's), reading the attested spellings of all of them (ἑπόμενες, beside the attested ἑπόμενη of επόμενος). When independently inflected nouns supply different explicit case-and-number cells, their whole-paradigm monotonic counts select the primary owner. Thus common feminine σκιά retains its own nominative, and σκατά the neuter plural of σκατό. Adjective or adverb readings of an inflected form do not invalidate its independent noun head. Nouns supplying the same cells retain the closest lexical stem, so a diminutive or synonym in another word's table does not acquire that word's declension. Finite verbs, names, and mixed headword analyses keep their existing selection rule. An independently defined adjective or adverb sharing the form does not reset an already selected noun owner: the noun cells still determine the primary and rare alternative readings. This refinement leaves a preferred adjective, finite verb, or name in its existing selection path.

A different explicit cell of another independently inflected noun remains at R with zero pooled count: χημεῖα, plural of χημείο, remains beside χημεία, and masculine genitive σκατᾶ beside neuter plural σκατά. These alternatives do not borrow the common homonym's frequency or provide evidence for raising another spelling's rating. A monotonic spelling has the letters, the accented vowel and any diaeresis of the polytonic one. The rest is decided as follows:

• the breathing of an initial vowel or ρ: the verb's attested polytonic forms with the same first two letters (ἔρθω, ἔρθει -> ἔρθεις); without them, the grc words with the longest prefix of the word, up to 7 letters, that at least 5 of them share, when 90% of those agree (ἀναρωτιέμαι from ἀνα-, εὑρέθηκα from εὑρε-, ἑστιάζω from ἑστια-). A prefix that many words share but do not agree on decides for the default, since its words are already further from the word than those of the longer one: εστι- is split by ἐστί, and εσ- below it, smooth almost throughout, would make ἐστιάζω. The default is the smooth breathing, the rough one on υ and ρ. An augment before a consonant is smooth (ἦβρε, not ἧβρε after ἥβη). On a diphthong the breathing goes on the second vowel;
• the type of the accent, by the Ancient rule that polytonic Modern Greek keeps, with syllables counted the Ancient way (ἤ-πι-ε): an accented antepenult takes the acute; an accented penult the circumflex when its vowel is long and the last syllable short (δῶσε, ποῦμε, ἦρθα, εἶσαι), else the acute; an accented last syllable the circumflex when its vowel is long (μπορεῖς, δοθεῖ, πιῶ, ἐπιζῶν), else the acute (πές). Only the aorist passive participle in -είς takes the acute on its long last syllable (γραφείς, beside the dependent γραφεῖς). η, ω and the diphthongs are long, ε and ο short, and a final αι or οι counts short. For α, ι and υ the verb's attested forms decide, where the type of their accent shows the length, if they have at least 3 tokens, the list's own floor: one token of γελᾶστε is a misspelling. Before a consonant of a perfective stem only forms with that consonant too count, since the long α of a contraction (γελᾶτε, κοιτᾶς) says nothing of the stem's (γελάστε, κοιτάξτε, κοιτάχτε); and a perfective form's acute (σκάσε) says nothing of an imperfective contraction (σκᾶς, σκᾶνε). Without such forms, the contracted endings are long: of the -άω verbs and of those filed under a polytonic -ῶ (μιλᾶμε, μιλᾶτε, μιλᾶς, μιλᾶνε), of the passive -άμαι and -ώμαι (θυμᾶμαι, αὐταπατᾶσαι) and of any imperative plural in -ᾶτε (ἐλᾶτε); any other vowel is short (κάνε, σκάσε). The -α of an imperative counts long (ζήτα, φεύγα). Monotonic writing leaves a monosyllable unaccented (πιω, πεις), so its accent goes on its last vowel (πιῶ, πιεῖς). The gerund keeps the acute, which the slice writes 283 times to the circumflex's 241 in the paradigms' gerunds (μιλώντας, ζητῶντας);
• for a noun, adjective, pronoun or name, the same rule, with the length of a last syllable that its vowel does not show taken as Ancient Greek has it. A final -ι is long (the neuters from -ιον keep their acute: συκώτι, γιαούρτι). A feminine singular -α is long (χώρα, καρδιά) except after σσ, ττ, ζ or a diphthong and ρ (γλῶσσα, μοῖρα) and in the -εια of an adjective in -ύς (ταχεῖα), and a feminine genitive -ας is long; a neuter plural, a masculine or any other -α is short (σχολεῖα, μῆνας, παιδιά), and a noun in -μα is neuter when its tags do not say (νῆμα). An accented last syllable takes the acute (ψυχή, καλοί, κινητό), and the circumflex, when it is long, only in the genitive (ψυχῆς, καλοῦ, καρδιᾶς, καρδιῶν, δικιᾶς), in a word of one syllable, in an adverb (ἀκριβῶς), and in the endings Ancient Greek contracted: the nouns in -άς, -ούς and -ού in every case (μπαμπᾶς, μπαμπᾶ, παπποῦς, ἀλεποῦ), the plural -εῖς of a noun or adjective (γονεῖς, ἀληθεῖς, ταχεῖς, but not an aorist passive participle filed under itself, γραφείς), and every ending of an adjective in -ής with a neuter in -ές but the nominative singular (ἀληθῆ, ἀληθοῦς). An adverb's -ά is short (ξανά). A form Wiktionary files with no cell, in -ού or -ών, is a genitive (ἐθνικοῦ, πνευματικῶν). The length of an α, ι or υ before the last syllable comes from the word's attested forms with the same letters through the next one, as for a verb, and is short without them. A name keeps its capital (Λονδίνο, Ἰούνιο);
• the iota subscript: the verb's attested forms with at least 3 tokens carry it over to the same vowel before the same letter (σῴσει beside σῴσῃ, but ἀγαπάει beside ἀγαπᾷ), and the contracted -ᾶς and -ᾶ take it as most of the slice's tokens of such forms do (69% and 83%: ἀγαπᾷς), except in a monosyllable (πᾶς).

The texts write the subjunctive of the second and third person singular both as the indicative (νὰ πάρεις, νὰ πάρει) and the traditional way (νὰ πάρῃς, νὰ πάρῃ), so both are generated, for the dependent forms, for the present forms that serve as their imperfective subjunctive, and for any form in -ει or -εις that Wiktionary tags only as a form of its verb (ξαναδῇ, ὁδηγῇς). Monotonic writing spells the subjunctive -ει(ς) and writes it -η(ς) only by mistake, a few times in a hundred; where it has the traditional spelling's letters and accent 50 times and a tenth as often as the form, they are another word's, and the twin is left out (the noun υπολογιστή, 12,664 tokens, beside the subjunctive ὑπολογιστῇ).

A generated spelling is listed only where neither grc nor the attested list has a lowercase spelling of its monotonic form (a diaeresis aside: χάιδεψε is the attested χάϊδεψε; a name such as Δροῦσε covers no δροῦσε), so it fills a gap. A Katharevousa dative of a noun, adjective or pronoun neither covers nor competes with the plain spelling Modern Greek writes (κέντρῳ beside κέντρο), while a verb's subscript is its contraction's (σταματᾷ) and competes like any other spelling. Other spellings of its letters, in the attested list, in grc or generated before it, compete with it for the same typed letters by the cell each spells, a cell being a word's form (_stops): two cells of the same letters are two forms, and both go in for the context to tell apart, whatever their counts (the imperative περπάτα and the present περπατᾷ, the noun τηλέφωνα and the imperative τηλεφώνα; ξεκινᾷ beside the attested ξεκίνα). Two spellings of the same cell are one form spelled two ways, and the generated one gives way to the attested one (ἀνέβουμε to ἀνεβοῦμε), to one generated before it, which monotonic text has at least as often (πεθάναν, 61 tokens, to πέθαναν, 8,404), and to a grc spelling whose monotonic form monotonic text has at least as often as its own (φάκελους, 59, to φακέλους, 2,712; χειροῦργος, from the misaccented χειρούργος, 1,286, to χειρουργός, 1,643). A noun's or adjective's cell is its case and number, the same under the noun and the adjective of one word (ὑπάλληλους gives way to ὑπαλλήλους), and another word of the same letters is another form (the adjective ἔνοχη beside the noun ἐνοχή). A spelling whose monotonic form is in no paradigm (an Ancient word, a word Wiktionary lacks) stops a generated one only where monotonic text writes it twice as often (RIVAL_RATIO), so καταλαβεῖς, of καταλαβεύς "clamp", leaves καταλάβεις, and a subjunctive twin, whose letters monotonic text writes only by mistake, gives way to any spelling of its letters it writes twice as often (ἐνημερώσῃ to the noun ἐνημέρωση, 3,970). The generated spellings come with the one whose own letters and accent monotonic text writes most first. A generated spelling must pass the list's orthography rules and recorded reviews, and each oxytone gets its contextual twin; a loanword may end in a consonant (μπάσκετ, μοτέλ, ἀλκοόλ), unless it is a word of grc or the list that lost its last vowel. Normally, a capitalized form (a name, a month) is listed where no spelling of its monotonic form is, capitalized or in lowercase (a spelling in -ῃ too: γράψῃ keeps out the surname Γράψη), a dative in -ᾳ or -ῳ aside (grc's Ῥωσίᾳ and ῥωσίᾳ leave Ῥωσία), where it would not give way as its lowercase spelling does (Ἅγιους, of the page Άγιος, to grc's ἁγίους), and then only with a mark monotonic writing drops (Ἰούνιο, Ὑόρκης, Ῥωσία), where monotonic text has it often enough for M (Λονδίνο, 14,283 tokens; Τζένη, 2,315), or where it has two of its declined forms, as it has a Greek word's (Δανία, 676, and Δανίας, 214), so that a keyboard takes it for a word rather than respell it (Λονδίνῳ, a dative of the attested list) or leave it bare. A rarer name in one form (Τζίμα, 85), mostly a foreign one, is right as typed, and in the list it would only draw names typed near it to itself (τζιμ, typed without an accent, corrected to Τζίμα). A generated name is R, since its lowercased count is shared with the words of its letters. An uncounted lowercase Ancient spelling (fr:X) alone does not cover a counted capital noun or name with an independently defined Modern head and two distinct, explicitly identified case-and-number forms. Literal capital coverage and counted lowercase coverage (C, M, or R) retain their guards. An Ancient spelling a review found to have the letters of the word Modern Greek means by them is left out of the list, is no evidence for a generated spelling and covers none, and is marked mg:avoid where grc has it (MG_LETTERS_OF_ANOTHER_WORD): εὑρῶ, the subjunctive of εὑρίσκω (5 tokens in the slice), beside the currency εὐρώ (4,311 monotonic tokens). It is written form<TAB>fr:B mg:generated, with the bucket B of its monotonic form's count (above), so a reader can tell it from the attested spellings; a Hunspell reader that does not know the field accepts it like any entry. The .version counts the attested and generated entries and their buckets, and records the sha256 of the paradigms and the frequencies (verb_paradigms_sha256, form_frequencies_sha256). Without those files the exporter says so and generates nothing, and --no-generated-verbs leaves the generated forms out. In the current export, 80,678 candidate analyses are covered by grc or the attested list, 2,219 remain beside a spelling of the same letters, and 1,234 give way to one. The 49,344 generated entries cover verbs, nouns, adjectives, names, standalone participles, and other source cells, including subjunctive twins, contextual twins, and plain nominal twins. Of the generated entries, 274 are in C and 4,950 in M.

The list leaves out what grc accepts, so the commonest Modern Greek words that keep their Ancient spelling (θέση, πόδια, κοίτα, ἀγορά) had no Modern Greek rating: a keyboard ranking the readings of Modern Greek text by the list's buckets saw only grc's, and preferred θέσῃ, rated from the monotonic θέσει. The list now rates them with lines of their own, θέση fr:C mg:shared (shared_lines): a grc spelling the slice attests takes the bucket an attested entry would, and a generated spelling grc has as written the bucket of its monotonic form's count, unless the slice writes that form otherwise (ἐμᾶς, not the generated ἐμάς). The current list has 6,651 shared rows (1,422 C, 5,105 M, 124 R). Low-frequency alternatives remain at R. These rows rate spellings already in the Ancient dictionary; they are no sign that a text is Modern Greek and are not additional entries: a reader that merges the two lists keeps grc's entry and reads the field to use the rating.

A spelling in -ῃ or -ῃς (θέσῃ, ἔρθῃς), a traditional subjunctive, is rated by its own use (subscript_count). Monotonic text writes it -ει or -εις, as it writes the indicative, the perfect and the dative of those letters (θα θέσει, έχει θέσει, εν θέσει), so the spelling takes only the part of that form's count which the slice writes with the subscript, (n + 1) / (n + m + 2) of it by the rule of succession, where n counts the slice's tokens of the spelling and m those of its -ει spellings: θέσῃ, 19 tokens beside 19 of θέσει, takes half of θέσει's 2,110 and is M, below θέση's C; σταματήσῃ, 21 beside 8, takes 73% of 25,119 and is C; a generated twin the slice has in neither spelling takes half. The whole -ει count rated θέσῃ before. The slice's count alone, or the monotonic count of the spelling's own letters, which monotonic text writes only by mistake, rates almost every such spelling R: measured as below, that broke 12 dev words on the author folds in the Automatic register, against 1 now, and lost the traditional subjunctive in mixed text (πρὶν σταματήσῃ, νὰ ἀποφασίσῃ left bare).

eval/eval_mg_paradigms.py measures reconstruction of held-out source spellings. It holds out, in turn, every form the slice attests at least 3 times with one spelling taking more than half its tokens (a capitalized form's capitalized tokens inside a sentence), generates it with that form and its subjunctive spelling left out of its word's evidence, and compares the two, contextual grave and acute counted as one and a capital as its small letter. The 81 forms the generator declines (an unaccented word of several syllables, a clitic of one) are left out:

Verb class or part of speech	Spelling	Of them, forms with 50 monotonic tokens (those generated)	Subjunctive spelling
verbs in -ω (γράφω)	99.0% of 5,027	99.2% of 3,416	91.1% of 1,058
-άω (μιλάω)	95.5% of 1,034	95.8% of 781	93.6% of 157
-ώ, -είς (μπορώ)	98.1% of 565	99.5% of 378	86.7% of 181
-μαι (έρχομαι)	96.4% of 716	97.3% of 444	87.3% of 118
other verbs	92.4% of 118	98.5% of 66	87.5% of 32
all verbs	98.1% of 7,460	98.5% of 5,085	90.5% of 1,546
nouns	96.5% of 11,871	96.7% of 7,444
adjectives	97.9% of 5,967	97.8% of 3,603
adverbs	93.5% of 864	92.9% of 396
pronouns	90.1% of 152	90.4% of 114
names	89.8% of 1,195	83.7% of 338
standalone participles	89.1% of 119	81.8% of 44
all other parts of speech	96.1% of 20,385	96.2% of 12,106

By cell, the past is right 99.9% of the time, the imperfect 99.3%, the present 98.9%, the imperative 99.0%, participles 96.2% and dependent forms 96.7%. Examples of competing spellings include the Katharevousa participle γραφεὶς beside the dependent γραφεῖς, ζητῶντας beside ζητώντας, μιλᾶς beside μιλᾷς, the nouns γραφή, λάμψη and δύση beside the subjunctives γραφῇ, λάμψῃ and δύσῃ, the relative ὃν beside the participle ὄν, and subjunctives that some texts write without the subscript (κρύβης). Among the nouns and adjectives they are the same letters in another cell (the genitive μαθητῆ beside the accusative μαθητή, the adjective καστανά beside the genitive καστανᾶ), Ancient datives the slice prefers (ἀφοβίᾳ, προτεραίᾳ), and words whose long vowels polytonic Modern Greek writes both ways (σαπούνι and σαποῦνι, κολώνα and κολῶνα, βοριάς and βασιλιᾶς). The names are compared with the slice's capitalized spellings, which are often another word at the start of a line.

The forms the list generates are the ones the slice does not attest, so a hand-checked sample of them gives a second number: eval/mg_generated_sample.tsv holds 356 generated verb spellings, stratified by verb class and cell, and 379 generated noun and adjective spellings, stratified by part of speech, case and number and fr: bucket, each checked for its breathing, accent type and subscript. Of the historical sample, 723 spellings remain generated verbatim, all with a prior correct judgment. One other sample, μαρή, is supplied by grc; the Modern list supplies its contextual grave μαρὴ. The other 11 samples are no longer generated. eval/eval_mg_paradigms.py reports the sample's membership and prior judgments on the current list.

The language model's dev split is per sentence, so a dev sentence's document and author also supply training sentences. eval/eval_mg_polytonic.py measures how many of the 34,315 words in its clean dev sentences the dictionary accepts, with the list built four ways, each with its generated forms read from its own sources:

Dictionary	Accepted, attested words only	Accepted, with generated forms	Out-of-dictionary words removed
grc alone	81.6%
grc + list from the training split	92.2%	93.3%	63%
grc + list without the documents holding a dev sentence	88.8%	91.3%	53%
grc + list without each dev word's author (5 author folds)	91.0%	92.5%	59%
grc + list without another author fold (control)	91.9%	93.0%	62%

The authors' own texts account for about 0.5 points of the 11.7 the list adds. The generated forms add 1.0 to 2.5 points, the most where the list has the fewest attested words. --write-lists DIR writes the author-fold lists, each with the mg:avoid lines and fr:R caps read from its own authors and the verb forms generated from their spellings, for a keyboard harness. The keyboard measurements below retain the earlier list and engine snapshots described in each comparison. One such measurement was made with a keyboard (not part of Dilemma) that has a Modern Greek register, offers the list's words only in Modern Greek text and reads the mg:avoid marks there. The dev sentences were typed as bare letters, with each author's fold held out from the list and its marks. Of the 2,415 sentences, 149 (2,493 words) are monotonic in the gold text itself, so they are left out; on the other 36,129 words the keyboard made these errors:

Register	Without mg:avoid	With mg:avoid	Fixed	Broken
Modern	5,248	4,965	332	49
Automatic	7,210	7,088	136	14

With the monotonic sentences counted as well, the 38,622 words give 6,210 and 5,943 in the Modern register, 8,250 and 8,137 in the Automatic one. Two earlier mark rules gave the same within a word: a share of all the letters' tokens with the lexicon's different-word check (207 lines on the same list) 4,966 and 7,089, and the same share with the respelling test (211 lines) 4,965 and 7,088. The weak forms carry nearly all of the gain: 126 με, 30 σε, 49 που and 4 πως fixed, against 5 με, 6 σε, 2 που and 4 πως that were correct enclitics (ἄφησέ με, οὕτω πως), 3 πῶς written πὼς, and 13 of the older relative ποῦ written ποὺ. On 474,220 words of held-out Ancient Greek sentences, the keyboard's Ancient and Automatic registers gave the same words with the list as without it.

A later build of the same keyboard measured the generated verb forms and the monotonic buckets against the attested words alone, as the earlier lists had them. On the author folds above (all 38,622 words), its errors went from 6,661 to 6,457 in the Modern register (209 words fixed, 5 broken) and from 8,014 to 7,396 in the Automatic one (634 fixed, 16 broken); the C and M buckets let it read more text as Modern Greek there, and write letters only a frequent list word has as that word. The Ancient register gave the same words. Typed as bare letters after θα and να, the active dependent forms of all six persons and the present second and third person singular of the 400 verbs with the most monotonic tokens (3,562 cases, each form at least 50 tokens) came out as the intended word 53.8% of the time before and 99.1% after, in the Modern and Automatic registers alike. Of 1,440 imperatives typed alone (360 forms of the same verbs, opening a sentence or after Ἄκου,, before ! or .), the Modern register wrote 31.3% as intended before and 96.0% after, the Automatic one 13.8% and 44.0%, none lost. On the held-out Ancient Greek sentences and passages the Ancient register gave the same words, byte for byte. In the Automatic register the held-out sentences gained 612 words and lost 14, all but 9 gained in the Modern Greek ones (8 Byzantine vernacular, 1 GLAUx), and the passages gained 287 and lost 17, in Modern Greek, Katharevousa and Byzantine vernacular but for 5 gained in patristic, capitalized and New Testament ones; a scholion apparatus that writes the Modern-looking νότο was read as Modern Greek once, so the question Τίνες after it came out Τινές.

The next version of that keyboard measured the list with the generated forms of every part of speech, against the list with generated verb forms only and against the keyboard's current list, all with each author's fold held out. On the author folds its errors went from 6,458 (verb forms only) and 6,663 (current list) to 6,249 in the Modern register (245 fixed and 36 broken against the first), and from 7,406 and 8,024 to 7,240 in the Automatic one (197 fixed, 31 broken). Most of the broken words are traditional subjunctives after νὰ and θὰ (νὰ προτιμήσῃ, νὰ ἐξετάσῃ), now written as the noun of the same letters (προτίμηση), and Katharevousa genitives (δυσαρεσκείας, αἰθούσας). The verb forms typed after θα and να came out as intended as often as before (99.1%); of the 1,440 imperatives typed alone, 6 were lost in the Modern register (95.5%) and 4 in the Automatic one (43.8%), where τηλεφώνα gives way to the noun τηλέφωνα, 24 times as common, and κούνα ranks no higher than the attested κουνᾶ. Of 550 everyday Modern Greek sentences typed in both registers, 127 came out better than with the verb forms only and none worse. The 5,000 commonest monotonic words, typed alone as bare letters in both registers, came out right 8,580 times of 10,000, against 7,625 with the verb forms only and 5,691 with the keyboard's current release and its list. Held-out Ancient Greek sentences and passages, and 1,280 further passages of Ancient, Koine, patristic, Byzantine and Katharevousa text, came out the same in the Ancient register, but for one Modern Greek sentence among the held-out ones in which ῥόζ, a loanword the list now has, made the keyboard respell the typed ροζ as ῥοῦ. In the Automatic register the held-out sentences gained 129 words and lost 28 and the passages gained 173 and lost 25, the losses again mostly subjunctives after νὰ.

Against that list, with the same keyboard build, the list as it is now fixed 8 words on the author folds in the Modern register and broke 1 (6,184 errors to 6,177), and fixed 3 and broke 1 in the Automatic one (7,190 to 7,188); the broken word, γιοὺς written γιὸς in the two folds whose authors attest it, was generated before from a stray row of the pairs, which lack the paradigm of γιος. The 10,000 commonest monotonic words, typed alone in both registers, came out right 103 more times and 8 fewer: the forms of ρωτάω, φακέλους, συναδέλφους, χειρουργός and κινέζικο gained, and ἐξάλλου, ὁποῖες, ἐνοχές and ἀξίες lost, beside which the rarer word of the same letters now goes in (ἔξαλλου, ὅποιες, ἔνοχες, ἄξιες, rated R against C and M) and is the keyboard's choice for a word alone at the start of a sentence. Of the 550 and 778 everyday sentences typed in both registers, 35 came out better and 1 worse; after θα and να 2 more verb forms came out as intended (99.2%), none fewer; of the imperatives typed alone 6 more and 4 fewer did in the Modern register, where τρεχάτε is now written as the present τρέχατε beside it (157 monotonic tokens against 129). In the Automatic register the held-out sentences gained 2 words and lost 2, the passages gained 11 and lost 1, and the further passages gained 7 and lost 4, three of them subjunctives that their own use now rates R, just below M (ἀπομακρυνθῇ, παρουσιάσῃ, ἐπικοινωνήσῃς), and ἐν τῇ Ῥωσίᾳ, now beside the generated Ῥωσία. The Ancient register gave the same words, byte for byte, on every set.

Boundary rewrites (grc_morph.json)

export_morphology.py writes build/hunspell/grc_morph.json, the table a keyboard uses to respell a word once the next word turns out to start with a vowel. nu lists the forms that take movable nu (Smyth 134): third-person singulars in -ε, in -σι, and in -στι (ἐστί, its compounds, crasis and prodelided spellings, but not the Doric ἐντί); third-person plurals in -σι, the subjunctive's included (ὦσι, epic ἐθέλῃσι); every dative plural in -σι, -ξι or -ψι, the article's and numerals' as much as a noun's or a participle's; and adverbs in -σι, and adverbs of place in -θε (Smyth 134 D), that the texts write with a nu in at least a tenth of their tokens (παντάπασιν, Ἀθήνησιν, πέρυσιν, πρόσθεν, ὄπισθεν, but not the deictic οὑτωσί or εἴθε). Where a spelling has another reading that cannot take nu, GLAUx's token counts decide: φέρε is an imperative 956 times and an unaugmented imperfect 45, so it is left out, while μέλλε, an imperative 14 times and an imperfect 7, stays, because the imperfect is spelled μέλλεν 17 more times before a vowel. A capitalized spelling is weighed with its lowercase word; a lowercase one only with its lowercase spellings, so a name's vocative (Κέλσε) does not decide the verb (κέλσε). Diorisis lists every candidate analysis on each token, so it shows that a spelling can take nu but does not vote, and the export refuses a GLAUx pairs file without counts. An unaugmented imperfect or thematic aorist is spelled like the imperative (κατένεγκε, ἐντύγχανε), and the taggers do not tell them apart, so such a spelling is left out unless the texts write it with its nu somewhere: GLAUx writes the nu on 99.6% of these past forms before a vowel, so one never written with it stood there only as the imperative. The augment is read after the verb's preverbs, from an accent it holds on the penult (συνεξεῦρε, Smyth 426), and for a stem that cannot show it (ἐνεγκ-) from the augmented spelling filed under the same verb or its simple verb. A liquid verb's first aorist (σήμηνε) has its imperative in -ον and stays. The aorist optative in -ειε takes the nu like any third-person singular in -ε (δόξειεν); the pluperfect in -ει is left out, since GLAUx writes it with the nu 11 times in 4,317. el maps a full form to its elided form, ending in U+1FBD. An elision is written only in the spelling the rules give: the short final vowel goes, an oxytone throws its accent back onto the new last vowel as an acute unless it is a preposition, conjunction or enclitic, and every other mark stays. It is written only when the tagged corpora or the lookup table attest exactly that spelling among the elided forms of a lemma that claims the full form, and the value takes the key's case. No entry is made for a final vowel that cannot elide: a long one, marked or not (η, ω, a circumflex, an iota subscript or adscript, the Attic accusative in -έᾱ of nouns in -εύς, the deictic -ί), the words Attic never elides (ὅτι, περί, πρό, ἄχρι, μέχρι; Smyth 72), a monosyllable not ending in ε, and the dative -ι and -σι, which elide only in epic. The tagged corpora say which spellings are datives; for a form they do not analyze (the lookup table's paradigm expansions, such as δμητῆρι), the forms of its lemma decide: a genitive in -ος beside it, the uncontracted -εϊ of an s-stem, or a third-declension stem that yields the -σι. A present participle's dative plural is also the verb's third person plural (κεύθουσι, τιμῶσι), so those keep their elision. el_modern holds the Modern Greek elisions in the same {full: elided} shape, read from the polytonic Modern Greek slice as the word list reads it: 39 pairs such as γιὰ -> γι᾽, στὸ -> στ᾽, τώρα -> τώρ᾽, ὅλα -> ὅλ᾽, ποὺ -> π᾽, εἶναι -> εἶν᾽, νὰ -> ν᾽, and κι -> κι᾽, which the slice writes for κι about half the times it precedes a vowel. An elided spelling qualifies with 10 tokens from 2 authors, and its full form is the commonest attested word that spells its letters and marks plus one vowel or vowel digraph (unaccented when the elided spelling keeps an accent, accented when it has lost it, as an oxytone's does) and is at least as common; an unaccented preposition's aspirated spelling (ἀφ᾽, καθ᾽) is left to the plain one. Only the pairs el lacks or spells otherwise are listed: 51 Modern Greek pairs agree with el (διὰ, ἀπὸ, μέσα, σὲ), and one differs (μιά is μί᾽ in el by the Ancient rule, μι᾽ here). A keyboard writing Modern Greek reads el with el_modern laid over it, and one writing Ancient Greek reads el alone. The pairs are kept out of el because a keyboard that elides automatically rewrites the word before every vowel, and Ancient Greek never elides the article τὸ or ὅτι.

Modern Greek elides optionally, and most of these elisions are the minority spelling, so el_modern_share gives each el_modern key its rate: of the times the slice writes a word the elided spelling stands for right before a vowel-initial word, the share it writes it elided (στ᾽ counts στὸ and στὰ together). A word the elided spelling stands for only through a long ending does not count: the article's τοῦ and the dative τῷ are not τ᾽. Only γιὰ (0.65) and στὸ (0.58) pass half, and κι is just under it (0.49); νὰ is at 0.29, τὸ at 0.20, τώρα at 0.06 and ὅτι at 0.01. With automatic elision off, a keyboard reads the table only to decide whether an apostrophe after a word is an elision mark. A reader that knows only nu and el ignores both keys; --no-modern-greek leaves them out, as the slice's parquet is needed to build them.

tests/test_export_morphology.py checks the built table against these rules.

Polytonic next-word prediction LM

train_lm.py + export_lm.py produce a compact next-word prediction language model, build/lm/grc_ngram.bin, aimed at the Tonos iOS keyboard extension for a QuickType-style suggestion strip over polytonic Greek.

This is a classical stupid-backoff trigram over GLAUx + Diorisis (~29.6M polytonic tokens, 1.48M sentences after a deterministic 2% dev split). Classical n-gram is a deliberate choice: inference is a couple of binary searches on mmap'd bytes (no ML runtime, no matmul), trivially under 1 ms per keystroke, and the artifact can be used directly from a keyboard extension without adding a Core ML dependency. A small neural LM would improve perplexity but cannot beat this on the cost side that keyboards are constrained by: cold-start memory and per-keystroke CPU.

Build

python train_lm.py --sanity         # few files per corpus, <1 min
python train_lm.py                  # full GLAUx + Diorisis, ~5 min
python train_lm.py --no-diorisis    # GLAUx-only baseline
python export_lm.py                 # writes grc_ngram.bin + .version, ~30 s
python eval_lm.py                   # writes eval_results.txt, ~90 s

export_lm.py reads the grc Hunspell dictionary (build/hunspell/grc_polytonic.dic, from export_hunspell.py --variant grc) to choose which out-of-vocabulary counts it carries (see below), so export the dictionary first, or pass --oov-min-count 0 to leave that table out. Re-export the LM after every rebuild of the grc dictionary: the table follows the dictionary it was chosen against, whose sha256 grc_ngram.version records. export_hunspell.py prints a note when the dictionary it wrote differs from that one, and

python export_lm.py --check-dictionary   # exit 1 when the LM needs re-exporting

checks the pair without exporting anything.

Corpus loaders live in train_lm.py (GLAUx, inline) and extract_diorisis_lm.py (Diorisis, beta-code to NFC). To add another Ancient Greek corpus, write a loader that yields (sentence_id, [<s>, ...NFC tokens..., </s>]) and append one entry to build_corpus_sources in train_lm.py; the counting, vocab, split, and eval stages do not need any changes.

The Diorisis pairs, frequency, language-model, and form-attestation readers share build/diorisis_source.py for quotation context. A terminal Beta Code apostrophe is removed as a duplicated closing quote only when the following punctuation repeats it and the same document independently contains the resulting form under every annotated lemma, part of speech, and morphology reading. Genuine elision, including ) notation, remains intact; unresolved quote/elision readings are preserved. The pairs builder writes the original and normalized forms, witness locations, source hashes, and generator revision to data/changes/diorisis_token_normalizations.json by default; --token-audit PATH chooses another audit destination. Each recorded change can be reversed with restore_source_beta. The same-cell rule also resolves a final apostrophe before plain sentence punctuation when the next sentence begins with a consonant. A vowel-initial continuation preserves possible elision. This distinguishes complete ἐπάιον followed by punctuation from the genuine elision ὤιετ᾽ εἶναι.

When normalization brings source cells together, the pair extractor retains their distinct part-of-speech and morphology readings. POS disambiguation counts each form/lemma/POS edge once per corpus, so additional morphology readings do not multiply that source's vote.

Preserving an unresolved corpus reading does not establish a new dictionary spelling. The Ancient export uses that audit to require an exact independent lexical head, a cited source-form review, or a closed grammatical exception before admitting an elision implicated in quotation punctuation. Corpus counts and generated lookup rows do not provide that exception. Spellings in the curated compatibility fixture remain eligible; explicit spelling rejections still take precedence. The export audit records the withheld forms and the hashes of its quotation and lexical evidence. Cited exact forms preserve genuine elision and printed prodelision spellings even when the same spelling also occurs with quotation punctuation in a corpus.

The GLAUx loader reads through the license filter in build/nc_filter.py, the same one the pair, frequency, attestation and tagger builders use: the 7 NonCommercial and 25 PROIEL-derived works are dropped whole, and the 40 Gorman-derived works contribute only their automatic sentences, since their manual sentences are the project's held-out gold. That removes 841,551 of the 17.0M tokens GLAUx contributes, so the counts above, measured before the filter reached this loader, fall by that much on the next rebuild. The filter reads GLAUx's metadata.txt, which the loader expects next to the --glaux directory; --glaux-metadata points it elsewhere, and a run stops rather than proceed without it.

The GLAUx pair builder also repairs malformed name lemmas when their source group contains one exact nominative singular independently defined as a name in the merged lexical evidence. It preserves ambiguous groups and competing readings. Run python build/build_glaux_pairs.py first; it writes corrected pairs and data/changes/glaux_name_lemmas.json with original rows, corrected rows, source fingerprints and the generator commit. The lemma-attestation and tagger-data builders validate and apply that same audit through --name-corrections PATH (the audit above by default). Use --lexical-heads PATH and --name-pairs PATH for alternate pinned inputs. The audit can reconstruct the original pairs with unapply_name_corrections. Pair extraction and audit loading require the source metadata.txt; missing or changed exclusion metadata stops the build, including statistics-only runs.

Output layout:

build/lm/
  grc_ngram.bin                 mmap-friendly binary, ~55 MB (v2)
  grc_ngram.version             semver + dilemma commit + vocab/context counts
  eval_results.txt              from eval_lm.py (combined dev split)
  eval_results_glaux.txt        eval restricted to GLAUx dev sentences
  eval_results_diorisis.txt     eval restricted to Diorisis dev sentences
  vocab.json                    intermediate from train_lm.py
  unigrams.json                 intermediate
  bigrams.tsv.gz                intermediate
  trigrams.tsv.gz               intermediate
  type_counts.tsv.gz            intermediate: training count of every type, in the vocabulary or not
  dev_sentences.txt             held-out dev set, deterministic split (seed 4242)
  dev_sentences_glaux.txt       same split, restricted to GLAUx sentences
  dev_sentences_diorisis.txt    same split, restricted to Diorisis sentences
  stats.json                    training corpus statistics

Artifact layout at a glance (full spec in the docstring of export_lm.py): a 128-byte little-endian header, a sorted UTF-8 vocab (binary-searchable by the Swift reader), a per-vocab unigram count column, and three flat sorted tables: unigram top-K, bigram top-K-per-w1, and trigram top-K-per-w1,w2. Each suggestion entry is 6 bytes: a u32 word id and an i16 fixed-point log probability (scale 1024). Lookup is a binary search on the trigram context (w1, w2); on miss, fall back to bigram (w1); on miss, fall back to the global unigram top-K. All tables are mmap'd and accessed by offset; nothing needs to be read into RAM.

Typeahead v2 (format_version = 2) adds two things versus v1:

• Independent top-K per table. Bigram contexts now store the top 30 continuations and trigram contexts the top 15, so the keyboard's mid-word prefix filter has enough candidates to work with even after a user has typed a character or two. The global unigram fallback stays at 10. The v2 reader refuses any file with format_version below 2, so old v1 binaries must be rebuilt.
• A per-vocab unigram count column, one u32 per vocab entry, indexed by sorted-vocab id. The Swift reader uses this to rank global prefix completions by corpus frequency when the current bigram/trigram top-K doesn't cover the user's stem, so the bar is never starved of useful entries even for rare context / common stem combinations.

Three later additions keep format_version at 2 and leave the header and every section above byte-for-byte what a v2 reader expects:

• Homograph spellings kept past the cut. A context row also keeps, after its top 30 (bigram) or 15 (trigram), every continuation seen at least 3 times there whose bare letters another vocabulary spelling shares with more than a grave, case or elision-glyph difference: ἢ and ἡ, ἐκείνῃ and ἐκείνη, αὐτοῦ and αὑτοῦ. A keyboard choosing between two spellings of the same letters then scores both from the same row instead of one by backoff. The first 30 / 15 entries of every row are unchanged, so next-word lists are too (20,000 of 20,000 held-out contexts gave the same six words). Mid-word completions change, because a kept spelling counts as a context entry and so ranks above completions found by scanning the vocabulary: on 4,000 held-out one- and two-letter prefixes, 1,207 top-3 lists changed and the word that followed was in them 1,485 times against 1,473. --keep-homograph-min-count 0 turns this off.
• Rows completed after a homograph spelling. To choose between ἢ and ἡ, a keyboard scores the word typed next after each of them, and the cut left almost every next word out of both bigram rows, so both readings fell back to that word's unigram probability and the choice was made without evidence. The bigram row of every homograph spelling now also keeps, past its cut, every continuation seen at least 3 times after it: 266,694 continuations in 1,491 rows, 1.6 MB. Trigram rows are cut as before, and the first 30 entries of every row, with them the next-word lists, are unchanged. --homograph-context-min-count 0 turns this off; with both minimums at 0 the exporter writes the plain top-K rows.
• Out-of-vocabulary unigram table. When no spelling of the typed letters is among the 80,000 vocabulary types, every spelling scored nothing and the dictionary's order decided. train_lm.py now writes the training count of every type, and the exporter appends the counts of the out-of-vocabulary types that are a dictionary spelling with a same-letter twin: 125,357 entries at 9 bytes each (a 64-bit FNV-1a hash of the spelling and a count byte), found through a 16-byte footer that ends the file. A type a reader resolves to a vocabulary id through its lowercase or its U+2019 elision spelling is left out, and a word's elision-glyph variants are summed under U+2019, the spelling a reader asks for. A reader that knows the footer scores such a spelling at the unigram tier, α² · c / N, as it scores a vocabulary word no row holds; one that does not ignores the trailing bytes. --oov-min-count 0 leaves the table out.

Measured by replaying the 26,906 held-out Ancient Greek sentences (474,220 words) of the dev split through a keyboard that types each word as bare letters and restores its marks, against the shipped 1.3.6 model: the homograph rows correct 1,972 words and break 549 (net +1,423); the table on top of them corrects 2,741 and breaks 424 (net +2,317). GLAUx and Diorisis annotate largely the same texts, so most dev sentences have a twin in the training split; on the 7,449 sentences with under 20% of their 5-grams in training (137,525 words) the two gains are +3.2 and +4.5 corrected words per 1,000. The completed rows, measured the same way against the model with the other two, correct 259 words and break 23 (net +236), and 104 and 2 on the low-overlap sentences (+0.74 per 1,000); typed with their punctuation, 265 and 20 (116 and 1). Every corpus of running text gains, and so does polytonic Modern Greek with each author held out.

Default knobs (defined in export_lm.py):

Knob	Value	Effect
vocab size	80,000	Reduces UNK target rate to 7.6% on the dev split
top-K uni	10	Global fallback depth
top-K bi	30	Per-bigram context depth (mid-word prefix filter wants this deep)
top-K tri	15	Per-trigram context depth
min bigram count	1	Keep every observed bigram
min trigram count	1	Keep every observed trigram...
min bigram count for trigram	3	...but only when the (w1, w2) bigram context is well attested. Rare contexts fall back to the bigram table at inference. This is the dominant size/quality knob.
keep homograph spellings	3	A homograph spelling seen this often in a context stays in its row past the top-K cut (0: off)
complete homograph rows	3	The bigram row of a homograph spelling keeps every continuation seen this often after it (0: off)
out-of-vocabulary count table	1	Minimum training count of a contested dictionary spelling the table carries (0: no table)

Current build: 61,892,317 bytes (59.0 MiB; 20.2 MB under xz -6), 80K vocab, ~1.1M trigram contexts, ~80K bigram contexts. The plain top-K rows are 56.9 MB of that, the kept homograph spellings 2.3 MB, the completed homograph rows 1.6 MB and the out-of-vocabulary table 1.1 MB.

Held-out evaluation, keyboard-realistic regime (exclude </s> and UNK targets):

Dev split	training data	sentences	preds scored	top-1	top-3	top-5	top-6
GLAUx only	GLAUx	19,578	315,385	13.56%	23.08%	27.75%	29.59%
GLAUx only	GLAUx+Diorisis	19,578	315,278	14.86%	28.14%	34.52%	36.56%
Diorisis only	GLAUx+Diorisis	10,672	188,199	15.17%	30.46%	37.63%	39.88%
Combined	GLAUx+Diorisis	30,250	503,477	14.97%	29.00%	35.68%	37.77%
Combined	v2 (bi 30/tri 15)	32,724	534,541	14.69%	28.30%	34.83%	37.12%

The v2 row reports on the newer combined corpus (GLAUx + Diorisis + Katharevousa Wikisource + Byzantine vernacular). Top-1 / top-3 / top-5 are unchanged from the v1 row's combined baseline (as expected: widening top-K per context does not move top-N for N smaller than K); the new number to look at is top-6 at 37.1%.

Adding Diorisis lifts top-3 accuracy on the identical held-out GLAUx sentences by +5.1 points and top-5 by +6.8 points; top-1 moves by +1.3 points. Dev sentences with identical sentence-id hashes across runs make this an apples-to-apples comparison.

Backoff level breakdown: ~74% of predictions hit the trigram table, ~26% fall back to bigrams, <0.01% fall to unigram. Perplexity (stupid backoff, α=0.4) is reported in eval_results.txt but is not a clean perplexity: because the binary only stores the top-K continuations per context, off-list targets are assigned a floor probability, which drives PPL up. The top-K accuracies are the real quality metric for this artifact; PPL is a sanity signal, not a headline number.

Expectations, honest:

• Top-1 ~14% is the expected order of magnitude for polytonic Ancient Greek. AG has highly variable constituent order, rich inflectional morphology, and a training corpus orders of magnitude smaller than English keyboard LMs. English QuickType-style predictors typically hit 15-30% top-1 on in-domain text; for AG, top-5 around 25-30% is a respectable baseline.
• UNK rate is 7.9% on the combined dev split even at 80K vocab. Polytonic inflection multiplies forms (a typical verb has 300+ forms), so long-tail coverage is inherently hard. Tonos can let the user type any form; the LM just won't propose it.
• Homographs are preserved. ἦ (past indicative) and ἤ (disjunction) remain distinct. No accent stripping is done at any stage.
• Diorisis beta code is converted to NFC polytonic via the betacode PyPI package. A small cleanup handles elision-after- consonant encoded as ) rather than ' (e.g. par) for παρ’). Elision-after-vowel (ou) = οὐ with smooth breathing) stays untouched.
• Corpus frequency keys preserve elision. GLAUx, Diorisis, PTA, and the OGC public-lexicon adapter use build/corpus_freq_key.py to strip accents while canonicalizing textual apostrophes and trailing combining psili to U+2019. merge_corpus_freq.py applies the same normalization again, so source-specific spellings cannot split counts across parallel keys. After a tokenizer change, regenerate that source's frequency artifact before merging; a lost apostrophe cannot be reconstructed from an old bare key.

Reader contract for Tonos: the binary format is versioned in the header (format_version = 2) and in grc_ngram.version. Any on-disk layout change will bump format_version. The sidecar version file also carries the dilemma commit hash, vocab size, and trigram / bigram context counts so a Tonos build can detect staleness and refuse mismatched artifacts. The Swift reader rejects any file whose format_version doesn't match the version it was compiled against; the v2 reader does not read v1 files (and vice versa).

Data

Sources and scale

Source	Forms	Notes
EN + EL Wiktionary (MG)	2.8M	From kaikki.org dumps
EN + EL Wiktionary (AG)	2.4M	From kaikki.org dumps
EL Wiktionary (Medieval)	6.9K	From kaikki.org dumps
LSJ + Sophocles expansion	Millions of AG additions	Current Wiktionary-Lua expansion plus marked, revision-pinned historical recovery
Perseus / AGDT (CC BY-SA 3.0 US)	81K	The 33 Greek AGDT works (Sophocles, Aeschylus, Homer, Hesiod, Herodotus, Thucydides, Plutarch, Polybius, Athenaeus); the original, not the NC UD release
GLAUx corpus	557K	17M tokens, 98.8% accuracy (Keersmaekers 2021)
Diorisis corpus	76K new	10M tokens, 91.4% accuracy (Vatri & McGillivray 2018)
HNC Golden Corpus	1K new	88K-token gold MG corpus (CLARIN:EL, openUnderPSI)
DGE headwords	52K	Headword filter coverage from Diccionario Griego-Espanol
LGPN names	44K	Proper noun coverage from Lexicon of Greek Personal Names
Perseus Digital Library headwords	176K	Headword filter from L&S, Pape, Bailly, etc.
Total lookup	14.5M	Deduped combined lookup rows (source rows above overlap)

All Wiktionary data is extracted automatically from kaikki.org JSONL dumps. LSJ and Sophocles expansions use wikitextprocessor to run Wiktionary's grc-decl and grc-conj Lua modules on headwords extracted from lexicon XML/TEI files.

The GLAUx corpus provides the largest single source of new form-lemma pairs outside Wiktionary. GLAUx is the primary corpus source due to its 98.8% lemma accuracy. The Diorisis corpus (Vatri & McGillivray, 2018; 10M tokens, Homer - 5th c. AD) is used as a secondary source: its 456K form-lemma pairs add 76K new entries not found in GLAUx. Because Diorisis has lower lemma accuracy (91.4%), its pairs are only added when they don't conflict with existing entries from Wiktionary, LSJ, or GLAUx. For form-frequency data, GLAUx and Diorisis are merged with the Patristic Text Archive and the Open Greek Corpus open-text rollup into the 94.0M-token corpus_freq.json (see Form-frequency corpora below).

We chose not to integrate one other large corpus:

• Opera Graeca Adnotata (OGA, 40M tokens): standoff PAULA XML format requires complex alignment code, and at 91.4% accuracy with 4x the size of Diorisis, the noise-to-signal ratio is worse for lookup purposes.
• Pedalion (5.8M tokens): smaller than GLAUx with similar classical-period coverage. Would add few forms not already covered by GLAUx + Wiktionary
  • LSJ, since the remaining lookup gaps are mostly Byzantine compounds not found in any classical corpus.

All three are CC BY-SA 4.0. Compound decomposition (added in v1.5) reduced the no-lookup-hit rate on DBBE from 4.4% to 2.5% by splitting compound words at linking vowels (ο/ι/υ), stripping known prefixes, and applying Byzantine-specific normalizations. The remaining 2.5% are forms where neither lookup, compound decomposition, nor the seq2seq model can recover the correct lemma.

Each form is indexed under its original, monotonic, and accent-stripped variants for fuzzy matching.

Form-frequency corpora and coverage diagnostics

data/corpus_freq.json is the merged form-frequency table consumed by rank_forms.py and vesuvius.py. It unions four sources (per-form counts summed, genre vectors element-wise added):

Source	Tokens	License	Genre breakdown
GLAUx	16.2M	CC BY-SA	yes (10 genres, lemma-aware)
Diorisis	10.2M	CC BY-SA	yes (10 genres, lemma-aware)
Patristic Text Archive	2.5M	CC BY-SA	bucket = religion
Open Greek Corpus rollup	65.1M	CC BY-SA	bucket = other
Total	94.0M tokens / 1.99M forms

The Open Greek Corpus rollup (its public_lexicon.tsv) covers First1KGreek, the corrected Patrologia Graeca OCR, Perseus canonical-greekLit, byzantium.gr historians and the Byzantine vernacular corpus, with OCR corrections and license filtering already applied. It replaces the three uncorrected per-repo builders (PG, First1KGreek, canonical-greekLit) that Dilemma used before.

Build:

python build/build_glaux_freq.py              # if not already built
python build/build_diorisis_freq.py
python build/build_pta_freq.py                # needs an Open Greek Corpus clone (sources/pta)
python build/build_cog_public_freq.py /path/to/pinned-ogc/data/public_lexicon.tsv
python build/merge_corpus_freq.py             # writes data/corpus_freq.json

The OGC adapter auto-detects the input checkout commit, hashes the generated public_lexicon.tsv, and records both values in cog_public_freq.json. merge_corpus_freq.py carries that source string into corpus_freq.json. The current rollup uses OGC commit f4062a50 and public-lexicon SHA-256 02d6de7181e8a108abe224081352d817c2064edf6234a51ecc63a5b8c83e6c2d.

The PTA TEI corpus is tokenised by build/tei_tokenize.py, which extracts text from <text> subtrees, drops apparatus / notes / bibliography, and splits on non-Greek characters with accent-stripped lowercase NFC keys -- the same shape GLAUx and Diorisis already use, so the merge is a straight element-wise sum. The Open Greek Corpus rollup arrives pre-tokenised in the same shape from that corpus's own build.

To inspect lookup coverage against the merged corpus run:

python build/coverage_report.py

As of the current pipeline, accent-relaxed ag_lookup.json coverage is 96.57% of running tokens in the merged 94.0M-token corpus and 36.77% of unique forms. coverage_report.py uses the same canonical elision key as the frequency builders, so U+2019 corpus apostrophes and U+1FBD lookup koronides compare correctly. None of the merged sources is lemmatised end-to-end by Dilemma, so they all contribute frequency and coverage signal but not new form-lemma pairs (lemma pairs come from Wiktionary plus the GLAUx/Diorisis/Perseus treebanks; the Gorman treebanks are held out as evaluation gold).

GlossAPI corpus audits

data/glossapi_corpora.json pins nine permissively licensed GlossAPI candidates by Hub commit, byte size, and SHA-256. The audit measures guess=False lookup coverage, unknown forms, conservative Unicode/OCR defects, and forms whose effective el and grc lookup rows disagree:

pip install -e '.[glossapi-audit]'
python scripts/audit_glossapi_coverage.py --source stream

For a resumable audit, add --sample-dir data/glossapi_samples. The saved counters are keyed by the source revision, artifact pins, reader, and sample limits; coverage is recomputed against the current Dilemma inputs. Reports record the code revision and the SHA-256 of the actual lookup and supporting tables. Generated reports and sample counters stay local and uncommitted.

Sampling assigns a separate token budget to each artifact and selected Parquet row group, and reads one group at a time. A row is a procedure, article, consultation, or volume depending on the source, so document counts are not directly comparable. The Parliament reader counts section once; its json_section field duplicates that text and adds serialization escapes. Unknown-form categories are triage hints, not confirmed dictionary omissions. The report distinguishes database language disagreements from disagreements that remain after removing citation spelling differences.

Frequency downsampling preserves the observed distribution instead of taking only its most frequent forms. The experiment retains per-source counts, source shares, and the checksum of the production frequency list. It still requires contextual evaluation before changing ambiguous-lemma ranking.

The October 20, 2026 manifest supersedes the unavailable OpenGov v2 repository with a verified pin of glossAPI/OpenGov_Deliberations. It retains the old pin for comparison and samples the replacement's archive and recent consultations separately. Equal artifact budgets balance those periods; they do not estimate their population proportions. Any inaccessible source is reported explicitly and keeps the frequency experiment incomplete.

Large sources can be deferred explicitly, without making a partial frequency experiment look complete:

python scripts/audit_glossapi_coverage.py --source stream \
  --exclude-corpus diavgeia

The generated coverage and frequency reports are diagnostics and never overwrite data/mg_freq.txt. Gutenberg, Ekklisiastika, and Archetai are audit-only: historical evidence must first pass through Open Greek Corpus provenance, language, OCR-quality, and deduplication gates. In particular, raw Ekklisiastika rubrics, joined tokens, and repeated hymn cycles must not feed language-model or frequency counts. OGC additionally blocks Ekklisiastika admission pending written permission for the underlying GOARCH text or an independently licensed replacement. A dataset-level license does not remove that gate.

Evaluate cleaned Modern Greek frequencies separately with:

python eval/eval_glossapi_frequency.py --lookup PATH_TO_PINNED_MG_LOOKUP

The evaluator retains the raw token windows and removes repeated lines, explicit field labels, and enumeration markers. It restores stress only in all-capitals words with one accented spelling in the pinned Modern lookup. Counts and reports remain local; --reuse-counts requires matching collector, source-manifest, lookup, and sampling pins. After reviewing an unchanged cleaning recipe, --collector-sha SHA256 can explicitly pin counters from an older collector while testing a new reranker. Runtime ranking inputs must match data/hf_manifest.json; missing attestation or POS tables stop the run. A smaller sanity run can use --corpus mitos --max-tokens 20000 --max-documents 32 --limit-eval 100 with separate --counts and --output paths.

The pinned UD Greek GDT development split selects among fixed mixture weights; the test split measures that choice and a predeclared 10% mixture. This is an experimental reranker of tied lookup candidates and source-recorded noun, verb, adjective, and adverb alternatives accepted by the existing POS validator. Gold POS is not used to select candidates, and preceding-article contexts are preserved. It reports paired gains/losses and compares against both production surface-frequency priors and the shipped runtime. Dictionary-mapped surface counts are uncertain lemma priors. They do not replace the runtime's annotated lemma attestation or data/mg_freq.txt. Adoption requires a positive test effect and replication; insufficient evidence leaves the experiment separate.

The October 20, 2026 run collected all six Modern sources at their declared caps. On the 9,243-token GDT development split it selected a 25% domain prior. On the separate 9,452-token test split, that mixture scored 8,200 correct lemmas, versus 8,193 for the production-frequency comparator and 8,213 for the shipped runtime. The gain over the frequency comparator did not exceed the runtime, so the mixture was not adopted and mg_freq.txt remains unchanged.

The GlossAPI variety classifier has a separate optional evaluation so GreekBERT does not become a Dilemma runtime dependency:

pip install -e '.[variety-eval]'
python eval/eval_glossapi_variety.py

Extraction sources

Form-lemma pairs come from three sources per Wiktionary entry:

1. Inflection tables (primary). Every cell in a verb conjugation or noun declension table becomes a form-lemma pair. Covers all tenses, moods, cases, numbers. Multi-form cells (e.g. Πηλείδᾱο / Πηλείδεω) are split into separate pairs.
2. form_of references. When a page says "form of X", that gives us an additional pair. Adds ~44K MG and ~6K AG pairs not found in inflection tables.
3. alt_of references. Alternative/variant spellings. Adds ~1K pairs.

HuggingFace Hub dataset

Generated data files (ranked form lists, form frequencies, scored lookups) are hosted at open-greek/dilemma-data on HuggingFace Hub. rank_forms.py downloads these by default instead of regenerating locally.

Available files:

File	Description
mg_ranked_forms.json	MG lemma to frequency-ranked form list
mg_form_freq.json	MG form to frequency count
mg_ranked_forms_verbose.json	MG ranked forms with per-corpus breakdown
mg_lookup_scored.json	MG scored lookup table
ag_ranked_forms.json	AG lemma to frequency-ranked form list
ag_form_freq.json	AG form to frequency count
ag_ranked_forms_verbose.json	AG ranked forms with per-corpus breakdown
med_ranked_forms.json	Medieval lemma to frequency-ranked form list
med_form_freq.json	Medieval form to frequency count
mg_polytonic_freq.json	MG polytonic form frequencies from Wikisource (38M tokens)
mg_polytonic_ranked.json	Monotonic MG form to ranked polytonic variants

This is separate from the main ciscoriordan/dilemma model repo, which hosts lookup.db, model weights, and other core data files.

Confidence tiers

Not all lookup entries are equally trustworthy. Forms from inflection tables are template-generated and may be wrong for irregular words. Each entry is scored on a 5-point scale:

Tier	Condition	MG count	AG count
5	Both EN + EL Wiktionary have a page for this form	63K	17K
4	EN Wiktionary has a page (no EL page)	22K	50K
3	EL Wiktionary has a page (no EN page)	1.05M	131K
2	Both EN + EL tables agree on the lemma	199K	64K
1	Single source, table-only	1.49M	2.21M

Higher confidence wins when two sources map the same form to different lemmas.

Dialect tagging

Ancient Greek forms from EN Wiktionary carry dialect tags extracted from inflection table headers (e.g. "Epic declension-1", "Attic contracted present"). These are propagated to every form in that table section:

Dialect	Tagged forms
Attic	262K
Epic	102K
Ionic	15K
Doric	10K
Koine	37K
Aeolic	4K
Laconian	672
Boeotian	555
Arcadocypriot	407

Quality controls

• Greek-only filter. All forms must contain only Greek Unicode characters (U+0370-03FF, U+1F00-1FFF, U+0300-036F). Removes Latin letters, digits, template artifacts.
• Chain-breaking. If form A maps to lemma B, and B maps to C, the chain is resolved to the real headword at build time. Fixes ~65K entries caused by accent-stripped key collisions and treebank convention differences.
• Pronoun cross-contamination. Greek Wiktionary dumps the entire pronoun paradigm table into each pronoun entry (e.g. εσύ lists εγώ as a "form"). Articles and determiners are restricted to headword-only. Pronoun forms that are headwords of other closed-class entries are skipped.
• Proper noun plural filter. EL Wiktionary generates plural forms for proper nouns via templates (413K junk entries like Αχιλλείς). These are skipped unless EN Wiktionary also lists them (which indicates a human editor intentionally added them, e.g. Έλληνες).
• Training pair validation. Every training pair's lemma must be a headword (maps to itself in the lookup). Pairs with non-headword lemmas are resolved to the real headword or dropped.

Related work

Vatri & McGillivray (2020) assessed the state of the art in Ancient Greek lemmatization via a blinded evaluation by expert readers. They found that methods using large lexica combined with POS tagging (CLTK backoff lemmatizer, Diorisis corpus) consistently outperformed pure ML approaches with smaller lexica. Dilemma follows the same principle: a large lookup table (14.5M forms) handles the vast majority of words, with a small model as fallback.

Celano (2025) presented state-of-the-art morphosyntactic parsing and lemmatization for Ancient Greek using GreTa and PhilTa models trained on the AGDT and OGA corpora. Best lemmatization F1 was 95.6% on classical text. These models require POS context; Dilemma operates on isolated words but benefits from a much larger form inventory.

Swaelens et al. (2024) tested lemmatization on unedited Byzantine Greek epigrams and found that classical accuracy (~95%) dropped 30+ points on Byzantine text due to itacism, crasis, and non-standard orthography. Their best hybrid method (transformer embeddings + dictionary lookup) reached 65.8%. Dilemma achieves 89.2% on the same dataset (91.8% with per-token POS, equiv-adjusted).

Swaelens et al. (2025) showed that multi-task learning (joint POS + morphology + lemma prediction) improved Byzantine lemmatization by ~9pp, reaching ~74-75%. They also demonstrated that subword-tokenizing transformers plateau on Byzantine Greek due to orthographic inconsistency, and called for character-level models as the next step. Dilemma's character-level encoder-decoder is this architecture, and its perfect tense oversampling and multi-task POS head are directly informed by their findings.

Known issues

These are inherent limitations or Wiktionary coverage gaps, not code bugs. Most can be fixed by editing the relevant Wiktionary entry, which will propagate into Dilemma via kaikki dumps.

Issue	Tokens	Notes
αὐτοῦ ambiguity	~200	Genuine lexical ambiguity: both an adverb ("here/there") and genitive of αὐτός. Resolved when POS context is available via lemmatize_pos().
μιν → ὅς	~340	Convention difference. Wiktionary maps μιν to the 3rd person pronoun. Perseus treebank uses μιν as its own lemma.
Lemma convention differences	~400	αὐτάρ vs ἀτάρ, κε vs ἄν - Wiktionary and Perseus use different citation forms for some Homeric particles. Handled by lemma equivalence groups for evaluation.

Future work

Per-lemma passive_subject hint for verbs

Apps that render English translations of Greek verb forms (e.g. flashcard or lexicon UIs showing ἐλέχθη → it was said) need to pick the right English subject pronoun for passive and mediopassive 3sg forms. The natural choice depends on whether the verb's typical patient is a person or a thing:

• thing-passive: "it was said", "it was done", "it was written"
• person-passive: "he/she was killed", "he/she was crowned", "he/she was baptized"

Selectional preferences of this kind are not exposed by morphology data or by raw LSJ glosses as a structured field. Consumers currently work around the gap with hand-curated allowlists keyed on the English verb base, which scales badly past ~150 entries.

A first-class fix would add a per-verb passive_subject tag (person | thing | either) to Dilemma's verb exports, sourced from Wiktionary semantic categories, gloss-text patterns, or a one-shot LLM annotation pass over the headword set.

Downstream MG paradigms

Modern Greek canonical paradigms are produced by Klisy rather than Dilemma. Klisy preserves its legacy collapsed inflections cells and adds a parallel inflections_voiced view with active_* and mediopassive_* keys. This gives voice-aware consumers stable cells without breaking frequency, export, or existing canonical consumers. Dilemma's paradigm generator remains scoped to Ancient Greek and does not duplicate Klisy's source-aware MG classifier.

Architecture

Small character-level encoder-decoder transformer (~4M parameters), trained from scratch on Greek lemmatization pairs. This is the standard architecture from SIGMORPHON morphological inflection shared tasks.

Component	Config
Encoder	3 transformer layers, 256 hidden, 4 heads
Decoder	3 transformer layers, 256 hidden, 4 heads
POS head	Linear (256 -> 10 tags), auxiliary task
Nominal head	Linear (256 -> 45 labels), gender/number/case
Verbal head	Linear (256 -> 69 labels), tense/mood/voice
FFN	512 dim
Vocabulary	~381 Greek characters + special tokens
Parameters	~4.2M
Inference	ONNX or PyTorch, beam search with headword filter

No pretrained weights - the model is small enough to train from scratch on 500K+ pairs in minutes. The character vocabulary covers all Greek Unicode ranges (monotonic, polytonic, extended). Three auxiliary classification heads (POS, nominal morphology, verbal morphology) share the encoder and improve representations via multi-task learning.

Why not ByT5?

An earlier version of Dilemma fine-tuned Google's ByT5-small (300M params). ByT5 processes raw UTF-8 bytes, so a 10-character Greek word becomes ~20 encoder steps. The custom transformer uses a Greek character vocabulary (~381 tokens), so the same word is ~10 steps. Combined with 75x fewer parameters:

	ByT5-small	Dilemma
Approach	Subword tokenizer (UTF-8 bytes)	Character vocabulary (~381 Greek chars)
Parameters	300M	4M
Training (3.5M pairs, 3 epochs)	~20 hours	~95 min
Dependencies	torch + transformers	torch only (or ONNX only)

Projects using Dilemma

• Lemma - Modern Greek dictionary app. Uses Dilemma's frequency-ranked inflection lists (rank_forms.py) and the MG equivalences derived from mg_lookup_scored.json to resolve looked-up words to their canonical Modern Greek headwords.
• Tonos - iOS polytonic Greek keyboard. Ships Dilemma's compact Hunspell exports (grc_polytonic.{dic,aff}) for spell-check inside the keyboard extension's tight memory budget, plus the trigram language model (grc_ngram.bin) for QuickType-style next-word prediction over polytonic Ancient Greek.

Credits

Dilemma builds on openly licensed lexica, treebanks, and corpora - Wiktionary, LSJ, the Sophocles lexicon, GLAUx, Diorisis, First1KGreek, the Patristic Text Archive, the Patrologia Graeca, the AGDT dependency treebanks, and more (the Gorman treebanks are deliberately not extracted from: they are the held-out gold corpus). The complete list, with authors, sources, and licenses, is in NOTICE.

License

Dilemma's source code is licensed MIT (full text in LICENSE). © 2026 Open Greek.

The data artifacts Dilemma builds - the lookup database, the trained model, the frequency tables, and the corpus-attestation databases - are derived from third-party data and remain under their upstream licenses; see NOTICE. Every source Dilemma ships is openly licensed; NonCommercial sources are never ingested, and as project policy PROIEL is not used at all, not even as evaluation data. PROIEL (CC BY-NC-SA, no permissive release) is dropped; the AGDT data is taken from the Perseus original (CC BY-SA 3.0 US) rather than the NonCommercial UD repackaging; Diorisis (CC BY 4.0) is kept; the Gorman treebanks (CC BY-SA 4.0) are deliberately never ingested - they are the project's held-out gold corpus (eval/eval_gorman_gold.py), and GLAUx sentences whose manual trees GLAUx sourced from Gorman are skipped for the same reason; the few NonCommercial GLAUx texts, the 25 GLAUx works whose trees derive from the PROIEL treebank (the NT books and Herodotus), and the per-file NonCommercial PTA texts are filtered out (build/nc_filter.py). The lemma model trains only on Wiktionary + GLAUx pairs. The Modern Greek tagger/parser is trained on UD_Greek-GUD + the CC BY-SA dialect treebanks, not the NonCommercial UD_Greek-GDT.