LatinCy Preprocess

PyPI version Python versions CI License: MIT

Latin text preprocessing: U/V normalization, long-s OCR correction, diacritics stripping, macron removal, and Beta Code → Unicode Greek conversion — plus Ancient Greek elision/accent normalization — with optional Rust acceleration and spaCy integration.

Consolidates latincy-uv and latincy-long-s into a single package.

Installation

pip install latincy-preprocess

For spaCy pipeline components:

pip install "latincy-preprocess[spacy]"

For Ancient Greek normalization:

pip install "latincy-preprocess[grc]"

For both:

pip install "latincy-preprocess[spacy,grc]"

(The quotes matter in zsh — the default shell on macOS — which otherwise treats the brackets as a glob pattern and fails with no matches found.)

Quick Start

from latincy_preprocess import normalize

normalize("Gallia eft omnis diuisa in partes tres")
# 'Gallia est omnis divisa in partes tres'

Per-Normalizer Usage

U/V Normalization

Converts u-only Latin spelling to proper u/v distinction using rule-based analysis:

from latincy_preprocess import normalize_uv

normalize_uv("Arma uirumque cano")
# 'Arma virumque cano'

Rules handle digraphs (qu), trigraphs (ngu), morphological exceptions (cui, fuit), positional context (initial, intervocalic, post-consonant), and case preservation.

Long-S OCR Correction

Corrects OCR errors where historical long-s (ſ) was misread as f, using n-gram frequency analysis from Latin treebank data:

from latincy_preprocess import LongSNormalizer

normalizer = LongSNormalizer()

word, rules = normalizer.normalize_word_full("ftatua")
# ('statua', [TransformationRule(...)])

text = normalizer.normalize_text_full("funt in fundamento reipublicae ftatua")
# 'sunt in fundamento reipublicae statua'

Two-pass strategy: Pass 1 applies high-confidence rules (impossible bigrams like ft, fp, fc). Pass 2 uses 4-gram frequency disambiguation for ambiguous word-initial f- patterns.

Diacritics and Macrons

from latincy_preprocess import strip_diacritics, strip_macrons

strip_macrons("ārma")
# 'arma'

strip_diacritics("λόγος")
# 'λογος'

Beta Code → Unicode Greek

Latin prose corpora often encode embedded Greek quotations as TLG/Perseus-style Beta Code. Convert it to polytonic Unicode (NFC):

from latincy_preprocess import beta_to_unicode

beta_to_unicode("zei/dwros a)/roura")
# 'ζείδωρος ἄρουρα'

Note: this transliterates every ASCII letter to Greek, so apply it only to isolated Beta Code spans, not mixed Latin/Greek text. Use is_betacode() to guard or segment input:

from latincy_preprocess import beta_to_unicode, is_betacode

span = "a)/nqrwpos"
clean = beta_to_unicode(span) if is_betacode(span) else span
# 'ἄνθρωπος'  —  Latin spans are left untouched

is_betacode() is a heuristic (Beta Code written with no diacritics is indistinguishable from Latin), but it reliably catches accented Greek and ignores ordinary Latin punctuation.

Ancient Greek Normalization

Requires the grc extra (pip install latincy-preprocess[grc]), which pulls in greek-normalisation.

Canonicalizes Ancient Greek for consistent tokenization and dictionary lookup — collapsing the many treebank/corpus encodings of the elision apostrophe to a single codepoint (U+2019), stripping lexicographic macron/breve marks, and folding grave → acute for lemma matching:

from latincy_preprocess.grc import (
    normalize_surface,
    normalize_norm,
    normalize_lookup_key,
    is_greek_word,
)

# Surface form: NFC + canonical elision apostrophe (U+2019)
normalize_surface("μυρίʼ")     # 'μυρί’'   (Tesserae U+02BC → U+2019)
normalize_surface("ἀλλ'")      # 'ἀλλ’'    (ASCII apostrophe → U+2019)

# NORM: restore closed-class elision, grave → acute, movable nu/sigma
normalize_norm("δʼ")           # 'δέ'
normalize_norm("ἀλλ’")         # 'ἀλλά'

# Lookup key: grave → acute + final-sigma folding for dictionary matching
normalize_lookup_key("φονὸς")  # 'φονός'
normalize_lookup_key("λογοσ")  # 'λογος'

# Guard: Greek letters + the canonical elision mark only
is_greek_word("μυρί’")         # True
is_greek_word("anthropos")     # False

Consolidates the previously independent normalization implementations across the LatinCy Greek pipelines into a single source of truth for the elision/accent standard.

spaCy Integration

Three pipeline components are available as spaCy factories:

Unified Preprocessor (recommended)

Chains long-s correction → U/V normalization in the correct order:

import spacy

nlp = spacy.blank("la")
nlp.add_pipe("latin_preprocessor")

doc = nlp("Gallia eft omnis diuisa in partes tres")
doc._.preprocessed          # 'Gallia est omnis divisa in partes tres'
doc[2]._.preprocessed       # 'est'
doc[2]._.preprocessed_lemma # normalized lemma

Either normalizer can be disabled:

nlp.add_pipe("latin_preprocessor", config={"uv": False})
nlp.add_pipe("latin_preprocessor", config={"long_s": False})

Standalone Components

nlp.add_pipe("uv_normalizer")
# doc._.uv_normalized, token._.uv_normalized, token._.uv_normalized_lemma

nlp.add_pipe("long_s_normalizer")
# doc._.long_s_normalized, token._.long_s_normalized

Rust Backend

When compiled with maturin, a Rust backend provides ~3x throughput for both normalizers. The backend is selected automatically:

from latincy_preprocess import backend

backend()  # 'rust' or 'python'

The Python backend is fully functional and used as the fallback.

Accuracy

U/V Normalization

Dataset Accuracy
Curated test set (100 sentences) 100%
UD Latin PROIEL (~21K u/v chars) ~98%
UD Latin Perseus (~18K u/v chars) ~97%

Long-S Correction

Pass 1 rules have a 0.00% false positive rate. Pass 2 disambiguation uses a protected allowlist of ~170 common Latin f- words (inline in long_s/_rules.py) plus n-gram frequency tables (JSON files in long_s/data/ngrams/).

Changelog

See CHANGELOG.md for release history.

Citation

@software{latincy_preprocess,
  title = {latincy-preprocess: Text Preprocessing for LatinCy Projects},
  author = {Burns, Patrick J.},
  year = {2026},
  url = {https://github.com/latincy/latincy-preprocess}
}

Acknowledgments

The betacode submodule adapts the Beta Code → Unicode conversion tables and algorithm from the Classical Language Toolkit (cltk.alphabet.grc.beta_to_unicode), used under the MIT License (Copyright © 2013 Classical Language Toolkit). It is reimplemented here on the Python standard library so the package remains dependency-free.

The grc submodule is built on James Tauber's greek-normalisation.

License

MIT

Project links

Data verified by PyPI on Aug 7, 2026
Data provided by the project maintainers, verified at the time the release was uploaded to PyPI.

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Newer release available (0.5.1)

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for diyclassics from gravatar.com diyclassics

Credits

Author: Patrick J. Burns

GitHub Statistics

Data verified by PyPI on Aug 7, 2026
The GitHub source repository was provided by the project maintainers and verified by PyPI at the time of upload. Stars, forks, and open issues/PRs are derived from that repository and have not been independently verified.

License expression

MIT
View SPDX License List

Requires

Python <3.15, >=3.10

Provides Extra

dev grc spacy

Tags

latin greek nlp normalization philology text-processing ocr

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

latincy_preprocess-0.5.0.tar.gz (172.9 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

latincy_preprocess-0.5.0-cp310-abi3-win_amd64.whl (366.4 kB view details)

Uploaded CPython 3.10+Windows x86-64

latincy_preprocess-0.5.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (500.2 kB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ x86-64

latincy_preprocess-0.5.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (491.1 kB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ ARM64

latincy_preprocess-0.5.0-cp310-abi3-macosx_11_0_arm64.whl (460.5 kB view details)

Uploaded CPython 3.10+macOS 11.0+ ARM64

latincy_preprocess-0.5.0-cp310-abi3-macosx_10_12_x86_64.whl (465.5 kB view details)

Uploaded CPython 3.10+macOS 10.12+ x86-64

File details

Details for the file latincy_preprocess-0.5.0.tar.gz.

File metadata

  • Download URL: latincy_preprocess-0.5.0.tar.gz
  • Upload date:
  • Size: 172.9 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for latincy_preprocess-0.5.0.tar.gz
Algorithm Hash digest
SHA256 a325fce93558cf5bb25f9949ebfd0fdb973d2b7b06bf2f65ba3dfa7bf7fa6711
MD5 683e16340aec34c27b947a242a0a2bfa
BLAKE2b-256 f9bb11e1490d55f46618c6336cfc18f24a8807a192d4b1b887737d642dcebddc

See more details on using hashes here.

Provenance

The following attestation bundles were made for latincy_preprocess-0.5.0.tar.gz:

Publisher: release.yml on latincy/latincy-preprocess

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file latincy_preprocess-0.5.0-cp310-abi3-win_amd64.whl.

File metadata

File hashes

Hashes for latincy_preprocess-0.5.0-cp310-abi3-win_amd64.whl
Algorithm Hash digest
SHA256 d5ba47495a153e3b1aae579947308294221e2a3bb46d2b50faca25a1b11176e7
MD5 1220c122bca25be76c3c995cfe5364ac
BLAKE2b-256 eb26fe90dbb1cb75d924e4809ca55fdf82536a8f85e6001e74c8c43103260fd5

See more details on using hashes here.

Provenance

The following attestation bundles were made for latincy_preprocess-0.5.0-cp310-abi3-win_amd64.whl:

Publisher: release.yml on latincy/latincy-preprocess

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file latincy_preprocess-0.5.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

File hashes

Hashes for latincy_preprocess-0.5.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 7ba5253bd81d126ff13bfb0b4512b026f5a8aca74b170dbab9e2539985c9852a
MD5 8cf34b0c838e804645228e48f7ff1c8f
BLAKE2b-256 9f646cf00a6055f9b6606d9f5e41b6ae0476ea08a94af013b804c95f2d9b5e05

See more details on using hashes here.

Provenance

The following attestation bundles were made for latincy_preprocess-0.5.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: release.yml on latincy/latincy-preprocess

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file latincy_preprocess-0.5.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

File hashes

Hashes for latincy_preprocess-0.5.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm Hash digest
SHA256 cd8edea37b079219afc3eb06cac001a973b9cc2169057f373dcb206cd2958c39
MD5 655510f4e977e8636066b7690a9dd4bf
BLAKE2b-256 1abaf96b8bb74b49a46b24fac614a48a27a6e6c637de9054ef706b4c29b288cb

See more details on using hashes here.

Provenance

The following attestation bundles were made for latincy_preprocess-0.5.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: release.yml on latincy/latincy-preprocess

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file latincy_preprocess-0.5.0-cp310-abi3-macosx_11_0_arm64.whl.

File metadata

File hashes

Hashes for latincy_preprocess-0.5.0-cp310-abi3-macosx_11_0_arm64.whl
Algorithm Hash digest
SHA256 6c90b09877e57f14272908f681656c8c6f8b7c07183384450822b6de7340f5d4
MD5 6e38b6ecdac08d61feb648a890339d75
BLAKE2b-256 333988f33d4f5c36ec1904dae4f1edcea6a7b68cef496f517136c865f6a056df

See more details on using hashes here.

Provenance

The following attestation bundles were made for latincy_preprocess-0.5.0-cp310-abi3-macosx_11_0_arm64.whl:

Publisher: release.yml on latincy/latincy-preprocess

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file latincy_preprocess-0.5.0-cp310-abi3-macosx_10_12_x86_64.whl.

File metadata

File hashes

Hashes for latincy_preprocess-0.5.0-cp310-abi3-macosx_10_12_x86_64.whl
Algorithm Hash digest
SHA256 beeb94b05277af9c0ee59c6453a4fbd9d07b0d25a6b0cf100f09cd425d3d6e3f
MD5 2bbcee9c268eec6bd0be8db660834921
BLAKE2b-256 8974056d0afb4bddf98fe53e63ba9cfcbb62330f25508fc9154f6e410a6d02a8

See more details on using hashes here.

Provenance

The following attestation bundles were made for latincy_preprocess-0.5.0-cp310-abi3-macosx_10_12_x86_64.whl:

Publisher: release.yml on latincy/latincy-preprocess

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Release history Release notifications | RSS feed

0.5.1

6 files

This release

0.5.0 This release

6 files

0.4.0

6 files

0.3.3

6 files

0.3.2

15 files

0.3.1

15 files

0.2.0

2 files

0.1.2

2 files

0.1.1

2 files

0.1.0

2 files

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page