latincy-preprocess 0.5.1

Latin text preprocessing: U/V normalization, long-s OCR correction, diacritics stripping, macron removal, and Beta Code → Unicode Greek conversion — plus Ancient Greek elision/accent normalization — with optional Rust acceleration and spaCy integration.

Consolidates latincy-uv and latincy-long-s into a single package.

Installation

pip install latincy-preprocess

For spaCy pipeline components:

pip install "latincy-preprocess[spacy]"

For Ancient Greek normalization:

pip install "latincy-preprocess[grc]"

For both:

pip install "latincy-preprocess[spacy,grc]"

(The quotes matter in zsh — the default shell on macOS — which otherwise treats the brackets as a glob pattern and fails with no matches found.)

Quick Start

from latincy_preprocess import normalize

normalize("Gallia eft omnis diuisa in partes tres")
# 'Gallia est omnis divisa in partes tres'

Per-Normalizer Usage

U/V Normalization

Converts u-only Latin spelling to proper u/v distinction using rule-based analysis:

from latincy_preprocess import normalize_uv

normalize_uv("Arma uirumque cano")
# 'Arma virumque cano'

Rules handle digraphs (qu), trigraphs (ngu), morphological exceptions (cui, fuit), positional context (initial, intervocalic, post-consonant), and case preservation.

Long-S OCR Correction

Corrects OCR errors where historical long-s (ſ) was misread as f, using n-gram frequency analysis from Latin treebank data:

from latincy_preprocess import LongSNormalizer

normalizer = LongSNormalizer()

word, rules = normalizer.normalize_word_full("ftatua")
# ('statua', [TransformationRule(...)])

text = normalizer.normalize_text_full("funt in fundamento reipublicae ftatua")
# 'sunt in fundamento reipublicae statua'

Two-pass strategy: Pass 1 applies high-confidence rules (impossible bigrams like ft, fp, fc). Pass 2 uses 4-gram frequency disambiguation for ambiguous word-initial f- patterns.

Diacritics and Macrons

from latincy_preprocess import strip_diacritics, strip_macrons

strip_macrons("ārma")
# 'arma'

strip_diacritics("λόγος")
# 'λογος'

Beta Code → Unicode Greek

Latin prose corpora often encode embedded Greek quotations as TLG/Perseus-style Beta Code. Convert it to polytonic Unicode (NFC):

from latincy_preprocess import beta_to_unicode

beta_to_unicode("zei/dwros a)/roura")
# 'ζείδωρος ἄρουρα'

Note: this transliterates every ASCII letter to Greek, so apply it only to isolated Beta Code spans, not mixed Latin/Greek text. Use is_betacode() to guard or segment input:

from latincy_preprocess import beta_to_unicode, is_betacode

span = "a)/nqrwpos"
clean = beta_to_unicode(span) if is_betacode(span) else span
# 'ἄνθρωπος'  —  Latin spans are left untouched

is_betacode() is a heuristic (Beta Code written with no diacritics is indistinguishable from Latin), but it reliably catches accented Greek and ignores ordinary Latin punctuation.

Ancient Greek Normalization

Requires the grc extra (pip install latincy-preprocess[grc]), which pulls in greek-normalisation.

Canonicalizes Ancient Greek for consistent tokenization and dictionary lookup — collapsing the many treebank/corpus encodings of the elision apostrophe to a single codepoint (U+2019), stripping lexicographic macron/breve marks, and folding grave → acute for lemma matching:

from latincy_preprocess.grc import (
    normalize_surface,
    normalize_norm,
    normalize_lookup_key,
    is_greek_word,
)

# Surface form: NFC + canonical elision apostrophe (U+2019)
normalize_surface("μυρίʼ")     # 'μυρί’'   (Tesserae U+02BC → U+2019)
normalize_surface("ἀλλ'")      # 'ἀλλ’'    (ASCII apostrophe → U+2019)

# NORM: restore closed-class elision, grave → acute, movable nu/sigma
normalize_norm("δʼ")           # 'δέ'
normalize_norm("ἀλλ’")         # 'ἀλλά'

# Lookup key: grave → acute + final-sigma folding for dictionary matching
normalize_lookup_key("φονὸς")  # 'φονός'
normalize_lookup_key("λογοσ")  # 'λογος'

# Guard: Greek letters + the canonical elision mark only
is_greek_word("μυρί’")         # True
is_greek_word("anthropos")     # False

Consolidates the previously independent normalization implementations across the LatinCy Greek pipelines into a single source of truth for the elision/accent standard.

spaCy Integration

Three pipeline components are available as spaCy factories:

Unified Preprocessor (recommended)

Chains long-s correction → U/V normalization in the correct order:

import spacy

nlp = spacy.blank("la")
nlp.add_pipe("latin_preprocessor")

doc = nlp("Gallia eft omnis diuisa in partes tres")
doc._.preprocessed          # 'Gallia est omnis divisa in partes tres'
doc[2]._.preprocessed       # 'est'
doc[2]._.preprocessed_lemma # normalized lemma

Either normalizer can be disabled:

nlp.add_pipe("latin_preprocessor", config={"uv": False})
nlp.add_pipe("latin_preprocessor", config={"long_s": False})

Standalone Components

nlp.add_pipe("uv_normalizer")
# doc._.uv_normalized, token._.uv_normalized, token._.uv_normalized_lemma

nlp.add_pipe("long_s_normalizer")
# doc._.long_s_normalized, token._.long_s_normalized

Rust Backend

When compiled with maturin, a Rust backend provides ~3x throughput for both normalizers. The backend is selected automatically:

from latincy_preprocess import backend

backend()  # 'rust' or 'python'

The Python backend is fully functional and used as the fallback.

Accuracy

U/V Normalization

Long-S Correction

Pass 1 rules have a 0.00% false positive rate. Pass 2 disambiguation uses a protected allowlist of ~170 common Latin f- words (inline in long_s/_rules.py) plus n-gram frequency tables (JSON files in long_s/data/ngrams/).

Changelog

See CHANGELOG.md for release history.

Citation

@software{latincy_preprocess,
  title = {latincy-preprocess: Text Preprocessing for LatinCy Projects},
  author = {Burns, Patrick J.},
  year = {2026},
  url = {https://github.com/latincy/latincy-preprocess}
}

Acknowledgments

The betacode submodule adapts the Beta Code → Unicode conversion tables and algorithm from the Classical Language Toolkit (cltk.alphabet.grc.beta_to_unicode), used under the MIT License (Copyright © 2013 Classical Language Toolkit). It is reimplemented here on the Python standard library so the package remains dependency-free.

The grc submodule is built on James Tauber's greek-normalisation.

License

MIT