latincy-lexicon 0.11.1

Whitaker's Words as LatinCy pipeline components for Latin NLP.

latincy-lexicon makes lexical data and morphological analysis available as spaCy pipeline components, designed for use with LatinCy language models. It is based on Whitaker's Words and the Perseus Project's Lewis & Short dictionary, though with LatinCy-specific corrections and modifications.

Quick Start

import spacy

nlp = spacy.load("la_core_web_lg")
nlp.add_pipe("whitakers_words")   # zero-config: bundled lexicon, no data files to build

doc = nlp("Poeta bonus carmina pulchra scribit.")

# Dictionary glosses — work out of the box
for token in doc:
    if token._.gloss:
        print(f"{token.text:12} {token._.gloss}")
# Poeta        poet
# bonus        good, honest, brave, noble, kind, pleasant, right, useful
# carmina      song/music
# pulchra      pretty
# scribit      write

Morphological analysis (token._.ww), reinflection, and paradigms use the analyzer, which you build once from the bundled data (latincy-lexicon build writes analyzer.json — see Data Setup):

nlp.add_pipe("paradigm_generator", config={"analyzer_path": "data/json/analyzer.json"})

# Reinflection: change morphological features, get the right Latin form
scribit = nlp("Poeta bonus carmina pulchra scribit.")[4]
print(scribit._.reinflect(Number="Plur"))    # scribunt
print(scribit._.reinflect(Tense="Imp"))      # scribebat
print(scribit._.reinflect(Voice="Pass"))     # scribitur

Features

• whitakers_words — Single pipeline component providing dictionary glosses (token._.lexicon), rule-based morphological analysis (token._.ww), and short definitions (token._.gloss)
• paradigm_generator — Generates complete inflectional paradigms for any lemma, with reinflection support (token._.paradigm, token._.reinflect)
• Standalone Generator API — Produce all inflected forms for a lemma, or build form-to-lemma lookup tables, without requiring spaCy
• POS-aware ranking — Uses upstream tagger/morphologizer output to rank ambiguous entries and parses
• Multi-signal disambiguation — Scores candidates using lemma match, morphological features, dependency labels, NER context, and dictionary frequency
• Clean glosses, originals preserved — dictionary glosses are stripped of Whitaker's inline formatting (pipe markers, - prefixes) and syntactic usage notes ((w/DAT), (ne + SUB = …), => cross-references); bibliographic citations are surfaced in a source_refs field, and the verbatim original senses are kept in gloss_orig on any entry the cleanup changed

Installation

pip install latincy-lexicon

Or for development:

git clone https://github.com/latincy/latincy-lexicon.git
cd latincy-lexicon
uv venv && source .venv/bin/activate
uv pip install -e ".[dev,spacy]"

Data Setup

Dictionary glosses need no setup — whitakers_words loads the bundled lexicon on first use (see Quick Start).

Morphological analysis (token._.ww), reinflection, and paradigm generation use the analyzer, which you build once from the bundled data:

latincy-lexicon build

This parses the bundled DICTLINE, INFLECTS, UNIQUES, and ADDONS files, applies patches (sum/esse, pronoun endings), reconstructs headwords, and writes analyzer.json and lexicon.json to data/json/. Pass analyzer_path="data/json/analyzer.json" to the components to enable these features.

Usage

import spacy

nlp = spacy.load("la_core_web_lg")
nlp.add_pipe("whitakers_words")   # bundled lexicon + analyzer, no data files needed

doc = nlp("Gallia est omnis divisa in partes tres.")

for token in doc:
    print(f"{token.text:12} {token._.gloss}")

Pipeline Components

whitakers_words

A single component that provides three token extensions:

• token._.lexicon — list of dictionary entries matching the token's lemma, with glosses, part of speech, principal parts, and age/frequency metadata. Each entry's glosses are cleaned (and sometimes corrected) glosses from Whitaker's inline formatting and syntactic notes; an entry may also carry source_refs (bibliographic citations such as L+S or Souter, extracted from the gloss text) and gloss_orig (the verbatim original Whitaker senses, present only when the cleanup changed them)
• token._.ww — full morphological parse list from the Words stem+ending engine, ranked by POS match, morphological features, dependency labels, NER context, and frequency
• token._.gloss — short definition from the top-ranked parse, with Whitaker's inline usage notes and citations removed

With no configuration the component loads both bundled data sources — no data files required:

• the bundled lexicon (build_lexicon()) for glosses + citation forms, keyed by the token's lemma, and
• the bundled analyzer (build_analyzer(), use_bundled_analyzer=True by default) — the WW stem+ending engine, built in memory from the same bundled data. Both are built on first use (~5 s each) and cached to ~/.cache/latincy-lexicon; nothing large ships in the wheel.

The analyzer matters even when you only want glosses: the lexicon is lemma-keyed, so when an upstream lemmatizer misses a form (e.g. contemplemur left as its own lemma), a lemma-only lookup finds nothing and the token would be dropped. The analyzer segments the surface form, the component looks the entry up by headword, and token._.gloss is recovered. token.lemma_ is never overwritten — the lemmatizer owns it; the corrected citation form surfaces via token._.lexicon[0]["headword"] and token._.ww[0]["lemma"].

Pass use_bundled_analyzer=False to restore the lighter lexicon-only mode (skips the analyzer build and its resident indexes, at the cost of dropping glosses on lemmatizer misses). Pass an explicit analyzer_path (from latincy-lexicon build) to use a prebuilt analyzer.json instead of the in-memory build, or lexicon_path to override the bundled lexicon. Best results when placed after all LatinCy pipeline components.

Macron filter (optional): pass macron_path pointing to a kaikki-derived macronized-form → UD morph index (built by latincy-words). When a macronized form is analyzed, the index constrains which parses are returned — e.g. puellā → ABL only. Falls back gracefully when a form is not in the index.

lewis_short

A dictionary-article overlay: looks the token's lemma up in the bundled Lewis & Short index and attaches ranked entry handles.

nlp.add_pipe("lewis_short")

doc = nlp("agit")
doc[0]._.lewis_short
# [{"id": "n9", "key": "ago", "orth": "ăgo", "pos": "v. a.", ...}]

Token extensions:

• token._.lewis_short — list of L&S entry handles for the token's lemma, homographs ranked best-first by POS compatibility. Handles are lean (id, key, orth, pos, gen, itype); pass config={"include_text": True} to inline the full article text, or fetch it on demand via nlp.get_pipe("lewis_short").get_entry(id).
• token._.lewis_short_senses — None by default. Pass config={"attach_senses": True} to populate it with the top-ranked entry's structured senses as a lean list of {"level", "n", "display_gloss"} dicts. Opt-in because the sense store is ~48 MB (loaded lazily on first use). Full sense detail — raw gloss, citations, sameAs linked-data ids — stays available via nlp.get_pipe("lewis_short").get_senses(id).

No sense selection is performed — all senses of the top-ranked entry are attached in dictionary order; picking the contextually right one is future WSD work.

paradigm_generator

Generates complete inflectional paradigms for Latin words. The inverse of the analyzer: given a lemma, it produces all inflected forms with UD morphological features.

nlp.add_pipe("paradigm_generator", config={
    "analyzer_path": "data/json/analyzer.json",
})

doc = nlp("Amat puellam.")
for token in doc:
    if token._.paradigm:
        print(f"{token.text}: {len(token._.paradigm)} forms")

Token extensions:

• token._.paradigm — list of inflected forms for the token's lemma, each with form, lemma, upos, feats (dict of UD features), and alternate (bool). None for punctuation or unknown lemmas. By default only the clean paradigm is exposed; pass config={"include_variants": True} to add_pipe to include alternate forms.
• token._.reinflect(**overrides) — returns a surface form matching the token's current morphology merged with the provided UD feature overrides, or None if no match exists.

doc = nlp("amat")
doc[0]._.reinflect(Number="Plur")           # "amant"
doc[0]._.reinflect(Tense="Imp")             # "amabat"
doc[0]._.reinflect(Tense="Imp", Number="Plur")  # "amabant"

Standalone Generator API

The Generator class can be used independently of spaCy:

from latincy_lexicon.generator import Generator

gen = Generator.from_json("data/json/analyzer.json")

# Generate all forms of a lemma. sort="paradigm" gives traditional
# pedagogical order (present → imperfect → future, …); the default
# sort="ud" preserves rule-traversal order for downstream NLP.
forms = gen.generate("amo", sort="paradigm")
rex_forms = gen.generate("rex", pos="N")     # nouns only (POS filter)

for f in forms[:5]:
    print(f"{f.form:15} {f.upos:6} {f.feats}")
# amo             VERB   Aspect=Imp|Mood=Ind|Number=Sing|Person=1|Tense=Pres|VerbForm=Fin|Voice=Act
# amas            VERB   Aspect=Imp|Mood=Ind|Number=Sing|Person=2|Tense=Pres|VerbForm=Fin|Voice=Act
# amat            VERB   Aspect=Imp|Mood=Ind|Number=Sing|Person=3|Tense=Pres|VerbForm=Fin|Voice=Act
# amamus          VERB   Aspect=Imp|Mood=Ind|Number=Plur|Person=1|Tense=Pres|VerbForm=Fin|Voice=Act
# amatis          VERB   Aspect=Imp|Mood=Ind|Number=Plur|Person=2|Tense=Pres|VerbForm=Fin|Voice=Act

# Build form→lemma lookup tables for batch processing
lookup = gen.to_lookup_dict(["rex", "puella"])
# {"rex": "rex", "regis": "rex", "regi": "rex", ..., "puella": "puella", ...}

Each Form has five fields: form (surface), lemma (citation), upos (UD POS), feats (UD feature string), and alternate (bool).

Canonical vs. alternate forms

By default generate() returns the clean textbook paradigm. Forms outside the standard paradigm — archaic/rare nominal forms (puellabus, puellai), redundant frequency siblings (regium), and proper-sense capitalizations (Deus under the common noun deus) — are flagged alternate=True and filtered out. Pass include_variants=True for the exhaustive set:

clean = gen.generate("puella")                         # textbook paradigm
full  = gen.generate("puella", include_variants=True)  # + puellabus, puellai, …

Every Form still carries the alternate flag, so a consumer can inspect or re-filter as needed. to_lookup_dict() uses the exhaustive set automatically, so form→lemma coverage stays maximal for NLP.

Note on verbs: verb forms are currently returned exhaustively even by default (include_variants does not yet filter them). The verb-alternate detector over-flags the standard forms of irregular verbs — esse, posse, the present system of eo, fers/fert — so filtering verb alternates is not yet reliable and is deferred to a future release. The alternate flag on verb forms should therefore be treated as advisory.

Acknowledgments

This project is built on Whitaker's Words, a Latin dictionary and morphological analysis program created by Colonel William A. Whitaker (USAF, Retired). The WORDS system — including its lexicon (DICTLINE), inflection tables (INFLECTS), and morphological analysis logic — is the foundation of latincy-lexicon. Whitaker made all parts of the WORDS system freely available for any purpose ("Permission is hereby freely given for any and all use of program and data.", cf. here); this project exists because of that generosity.

The WORDS data files used by this project are maintained at mk270/whitakers-words. Thank you to Martin Keegan for continuing Whitaker's work and sharing that work in the same spirit.

License

The original Python code in this project is released under the MIT License.

The Whitaker's Words data and analysis logic incorporated in this project are copyright William A. Whitaker (1936–2010) and distributed under his original permissive license (see LICENSE for full text).