latincy-vocab 0.4.0

Latin vocabulary list builder powered by LatinCy models/tools/datasets.

latincy-vocab takes Latin text, runs it through a LatinCy spaCy model, and returns structured vocabulary lists. Citation forms (principal parts, gender, etc.), POS markers, and dictionary glosses are sourced from latincy-lexicon (Whitaker's Words); latincy-vocab is the formatting and aggregation layer over it. Output can be sorted by frequency, reading order, or alphabetically, and exported as JSON or Markdown.

Installation

pip install latincy-vocab

You also need a LatinCy spaCy model. latincy-vocab defaults to la_core_web_lg (best accuracy for citation forms and lemmatization):

pip install "https://huggingface.co/latincy/la_core_web_lg/resolve/main/la_core_web_lg-3.9.8-py3-none-any.whl"

A lighter la_core_web_sm is also available (swap lg→sm in the URL); set PipelineConfig(spacy_model="la_core_web_sm") to use it.

Glosses and citation forms come from latincy-lexicon (Whitaker's Words), which is installed automatically as a dependency — no extra data files, environment variables, or sibling directories required.

Usage

from vocabbuilder import VocabPipeline

pipeline = VocabPipeline()

text = "agricolae in villa laborant et aquam portant"
vocab = pipeline.process(text)

# Reading order (first occurrence)
for entry in vocab.by_first_occurrence():
    print(entry.formatted())

Example output:

agricola, agricolae, m., farmer, cultivator, gardener, agriculturist
in, prep., in, on, at (space)
villa, villae, f., farm/country home/estate
laboro, laborare, laboravi, laboratum, v., work, labor
et, conj., and, and even
aqua, aquae, f., water
porto, portare, portavi, portatum, v., carry, bring

Sorting

vocab.by_frequency()       # most common first
vocab.by_alpha()           # alphabetical by lemma
vocab.by_first_occurrence() # reading order

Filtering

vocab.filter_pos({"NOUN", "VERB"})   # nouns and verbs only
vocab.filter_min_frequency(2)         # words appearing ≥ 2 times

Static word lists. Drop lemmas already covered by a reference list (e.g. the DCC Latin Core Vocabulary), or keep only lemmas on a list. Matching folds u/v/j and case, so lists in any orthography work:

vocab.filter_lemmas(exclude=dcc_core)   # keep only what a passage adds beyond the core
vocab.filter_lemmas(keep=my_wordlist)   # keep only lemmas on my list

Keyness measures. Keep the words distinctive to a passage and drop corpus-wide function words, using a keyness weight per lemma. filter_keyness accepts whatever representation you already have — a plain {lemma: weight} dict, a pandas Series or DataFrame, an (lemma, weight) iterable, or a scipy-sparse / numpy document-term matrix straight from scikit-learn's TfidfVectorizer (no pandas/scipy/sklearn dependency is added — they're duck-typed). A lemma absent from the weights counts as 0:

# The DH-standard path: a sparse DTM from TfidfVectorizer.
from sklearn.feature_extraction.text import TfidfVectorizer

tfidf = TfidfVectorizer()
dtm = tfidf.fit_transform(corpus)            # scipy sparse, (n_docs × n_terms)
names = tfidf.get_feature_names_out()
# Keep the 20 most distinctive words of document i:
vocab.filter_keyness(dtm, feature_names=names, document=i, top_n=20)

# A pandas DTM row (index = terms) needs no feature_names:
vocab.filter_keyness(dtm_df.loc["ep_6.16"], min_score=0.1)

# …or a plain mapping you built yourself:
vocab.filter_keyness({"pumex": 0.7, "ut": 0.01}, min_score=0.1)

Export

# Markdown glossary
print(vocab.to_markdown())

# JSON (all fields)
print(vocab.to_json())

Coverage gaps. Some content words the tagger keeps have no Whitaker's Words gloss — most often a proper noun the model tags as an adjective, e.g. Lavinia (from Laviniaque), for which Lavinius/-a/-um is simply absent from the dictionary. An entry with a blank definition is useless in a glossary, so to_markdown, to_dicts, and to_json omit gloss-less entries by default. Pass include_missing_gloss=True to keep them, and use vocab.missing_gloss (a VocabList view) to inspect exactly which lemmas fell through:

print(vocab.to_markdown())                          # gaps hidden
print(vocab.to_markdown(include_missing_gloss=True)) # gaps shown

for entry in vocab.missing_gloss:                    # the gaps themselves
    print(entry.lemma, entry.pos)

The lexicon-free path (PipelineConfig(use_glosses=False)), where every entry is intentionally gloss-less, is unaffected — nothing is treated as a gap and all entries render.

Entry fields

Each VocabEntry exposes:

Configuration

from vocabbuilder import VocabPipeline, PipelineConfig

config = PipelineConfig(
    spacy_model="la_core_web_lg",  # default; lighter: "la_core_web_sm"
    min_frequency=2,               # drop words seen only once
)
pipeline = VocabPipeline(config)

Proper names

Bare proper names (PROPN) are excluded by default — they belong to a separate NER/NEL channel, not a vocabulary list. But the model routinely tags genuine vocabulary as PROPN when a word doubles as a name — the textbook example is Musa (the Muse) vs. the ordinary noun musa, musae, f. "muse". The signal that such a token is a real lexical item and not a bare name is that Whitaker's Words glosses it, so a glossed PROPN (Musa, Roma, Gallia) is kept — with its citation form and gloss — while an unglossed one (Aquitani, Celtae) stays out. Set keep_glossed_propn=False for strict drop-all-PROPN behavior:

config = PipelineConfig(keep_glossed_propn=False)  # no PROPN in the list at all

Related packages

• latincy-lexicon — Whitaker's Words lexical data for LatinCy
• latincy-preprocess — Latin text preprocessing utilities

License

MIT