latincy-preprocess

LatinCy Preprocess

Latin text preprocessing: U/V normalization, long-s OCR correction, diacritics stripping, macron removal, and Beta Code → Unicode Greek conversion — plus Ancient Greek elision/accent normalization — with optional Rust acceleration and spaCy integration.

Consolidates latincy-uv and latincy-long-s into a single package.

Installation

pip install latincy-preprocess

For spaCy pipeline components:

pip install latincy-preprocess[spacy]

For Ancient Greek normalization:

pip install latincy-preprocess[grc]

Quick Start

from latincy_preprocess import normalize

normalize("Gallia eft omnis diuisa in partes tres")
# 'Gallia est omnis divisa in partes tres'

Per-Normalizer Usage

U/V Normalization

Converts u-only Latin spelling to proper u/v distinction using rule-based analysis:

from latincy_preprocess import normalize_uv

normalize_uv("Arma uirumque cano")
# 'Arma virumque cano'

Rules handle digraphs (qu), trigraphs (ngu), morphological exceptions (cui, fuit), positional context (initial, intervocalic, post-consonant), and case preservation.

Long-S OCR Correction

Corrects OCR errors where historical long-s (ſ) was misread as f, using n-gram frequency analysis from Latin treebank data:

from latincy_preprocess import LongSNormalizer

normalizer = LongSNormalizer()

word, rules = normalizer.normalize_word_full("ftatua")
# ('statua', [TransformationRule(...)])

text = normalizer.normalize_text_full("funt in fundamento reipublicae ftatua")
# 'sunt in fundamento reipublicae statua'

Two-pass strategy: Pass 1 applies high-confidence rules (impossible bigrams like ft, fp, fc). Pass 2 uses 4-gram frequency disambiguation for ambiguous word-initial f- patterns.

Diacritics and Macrons

from latincy_preprocess import strip_diacritics, strip_macrons

strip_macrons("ārma")
# 'arma'

strip_diacritics("λόγος")
# 'λογος'

Beta Code → Unicode Greek

Latin prose corpora often encode embedded Greek quotations as TLG/Perseus-style Beta Code. Convert it to polytonic Unicode (NFC):

from latincy_preprocess import beta_to_unicode

beta_to_unicode("zei/dwros a)/roura")
# 'ζείδωρος ἄρουρα'

Note: this transliterates every ASCII letter to Greek, so apply it only to isolated Beta Code spans, not mixed Latin/Greek text. Use is_betacode() to guard or segment input:

from latincy_preprocess import beta_to_unicode, is_betacode

span = "a)/nqrwpos"
clean = beta_to_unicode(span) if is_betacode(span) else span
# 'ἄνθρωπος'  —  Latin spans are left untouched

is_betacode() is a heuristic (Beta Code written with no diacritics is indistinguishable from Latin), but it reliably catches accented Greek and ignores ordinary Latin punctuation.

Ancient Greek Normalization

Requires the grc extra (pip install latincy-preprocess[grc]), which pulls in greek-normalisation.

Canonicalizes Ancient Greek for consistent tokenization and dictionary lookup — collapsing the many treebank/corpus encodings of the elision apostrophe to a single codepoint (U+2019), stripping lexicographic macron/breve marks, and folding grave → acute for lemma matching:

from latincy_preprocess.grc import (
    normalize_surface,
    normalize_norm,
    normalize_lookup_key,
    is_greek_word,
)

# Surface form: NFC + canonical elision apostrophe (U+2019)
normalize_surface("μυρίʼ")     # 'μυρί’'   (Tesserae U+02BC → U+2019)
normalize_surface("ἀλλ'")      # 'ἀλλ’'    (ASCII apostrophe → U+2019)

# NORM: restore closed-class elision, grave → acute, movable nu/sigma
normalize_norm("δʼ")           # 'δέ'
normalize_norm("ἀλλ’")         # 'ἀλλά'

# Lookup key: grave → acute + final-sigma folding for dictionary matching
normalize_lookup_key("φονὸς")  # 'φονός'
normalize_lookup_key("λογοσ")  # 'λογος'

# Guard: Greek letters + the canonical elision mark only
is_greek_word("μυρί’")         # True
is_greek_word("anthropos")     # False

Consolidates the previously independent normalization implementations across the LatinCy Greek pipelines into a single source of truth for the elision/accent standard.

spaCy Integration

Three pipeline components are available as spaCy factories:

Unified Preprocessor (recommended)

Chains long-s correction → U/V normalization in the correct order:

import spacy

nlp = spacy.blank("la")
nlp.add_pipe("latin_preprocessor")

doc = nlp("Gallia eft omnis diuisa in partes tres")
doc._.preprocessed          # 'Gallia est omnis divisa in partes tres'
doc[2]._.preprocessed       # 'est'
doc[2]._.preprocessed_lemma # normalized lemma

Either normalizer can be disabled:

nlp.add_pipe("latin_preprocessor", config={"uv": False})
nlp.add_pipe("latin_preprocessor", config={"long_s": False})

Standalone Components

nlp.add_pipe("uv_normalizer")
# doc._.uv_normalized, token._.uv_normalized, token._.uv_normalized_lemma

nlp.add_pipe("long_s_normalizer")
# doc._.long_s_normalized, token._.long_s_normalized

Rust Backend

When compiled with maturin, a Rust backend provides ~3x throughput for both normalizers. The backend is selected automatically:

from latincy_preprocess import backend

backend()  # 'rust' or 'python'

The Python backend is fully functional and used as the fallback.

Accuracy

U/V Normalization

Dataset	Accuracy
Curated test set (100 sentences)	100%
UD Latin PROIEL (~21K u/v chars)	~98%
UD Latin Perseus (~18K u/v chars)	~97%

Long-S Correction

Pass 1 rules have a 0.00% false positive rate. Pass 2 disambiguation uses a protected allowlist of ~170 common Latin f- words (inline in long_s/_rules.py) plus n-gram frequency tables (JSON files in long_s/data/ngrams/).

Changelog

See CHANGELOG.md for release history.

Citation

@software{latincy_preprocess,
  title = {latincy-preprocess: Text Preprocessing for LatinCy Projects},
  author = {Burns, Patrick J.},
  year = {2026},
  url = {https://github.com/latincy/latincy-preprocess}
}

Acknowledgments

The betacode submodule adapts the Beta Code → Unicode conversion tables and algorithm from the Classical Language Toolkit (cltk.alphabet.grc.beta_to_unicode), used under the MIT License (Copyright © 2013 Classical Language Toolkit). It is reimplemented here on the Python standard library so the package remains dependency-free.

The grc submodule is built on James Tauber's greek-normalisation.

License

MIT

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

latincy_preprocess-0.4.0.tar.gz (170.7 kB view details)

Uploaded Aug 3, 2026 Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

latincy_preprocess-0.4.0-cp310-abi3-win_amd64.whl (364.1 kB view details)

Uploaded Aug 3, 2026 CPython 3.10+Windows x86-64

latincy_preprocess-0.4.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (497.8 kB view details)

Uploaded Aug 3, 2026 CPython 3.10+manylinux: glibc 2.17+ x86-64

latincy_preprocess-0.4.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (488.8 kB view details)

Uploaded Aug 3, 2026 CPython 3.10+manylinux: glibc 2.17+ ARM64

latincy_preprocess-0.4.0-cp310-abi3-macosx_11_0_arm64.whl (458.1 kB view details)

Uploaded Aug 3, 2026 CPython 3.10+macOS 11.0+ ARM64

latincy_preprocess-0.4.0-cp310-abi3-macosx_10_12_x86_64.whl (463.1 kB view details)

Uploaded Aug 3, 2026 CPython 3.10+macOS 10.12+ x86-64

File details

Details for the file latincy_preprocess-0.4.0.tar.gz.

File metadata

Download URL: latincy_preprocess-0.4.0.tar.gz
Upload date: Aug 3, 2026
Size: 170.7 kB
Tags: Source
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for latincy_preprocess-0.4.0.tar.gz
Algorithm	Hash digest
SHA256	`f38298d420b0ccea991f18ddb1372ac689468754bc4e9ff17e18627319d28c6e`
MD5	`5733eb2d1824c29d443a7ad33105f2ac`
BLAKE2b-256	`dd4a9223f1a9a44abc969a264cd4b64406ab3d373a236fadec17adc975a79067`

See more details on using hashes here.

Provenance

The following attestation bundles were made for latincy_preprocess-0.4.0.tar.gz:

Publisher: release.yml on latincy/latincy-preprocess

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: latincy_preprocess-0.4.0.tar.gz
- Subject digest: f38298d420b0ccea991f18ddb1372ac689468754bc4e9ff17e18627319d28c6e
- Sigstore transparency entry: 2337355469
- Sigstore integration time: Aug 3, 2026
Source repository:
- Permalink: latincy/latincy-preprocess@6818d97b179ae2c8e5f771f45d8a96635e77e1fe
- Branch / Tag: refs/tags/v0.4.0
- Owner: https://github.com/latincy
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: release.yml@6818d97b179ae2c8e5f771f45d8a96635e77e1fe
- Trigger Event: push

File details

Details for the file latincy_preprocess-0.4.0-cp310-abi3-win_amd64.whl.

File metadata

Download URL: latincy_preprocess-0.4.0-cp310-abi3-win_amd64.whl
Upload date: Aug 3, 2026
Size: 364.1 kB
Tags: CPython 3.10+, Windows x86-64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for latincy_preprocess-0.4.0-cp310-abi3-win_amd64.whl
Algorithm	Hash digest
SHA256	`a77391a8cc1c7e12f33e88f2550f67041f6b7cda105c1ee28fc63477ec55b809`
MD5	`5bc938e106628a8c86fde7b8a7123d7e`
BLAKE2b-256	`63850169a96dfc51fc619089a5f10920a649e94e85ae05800ef139a5d4d1a73f`

See more details on using hashes here.

Provenance

The following attestation bundles were made for latincy_preprocess-0.4.0-cp310-abi3-win_amd64.whl:

Publisher: release.yml on latincy/latincy-preprocess

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: latincy_preprocess-0.4.0-cp310-abi3-win_amd64.whl
- Subject digest: a77391a8cc1c7e12f33e88f2550f67041f6b7cda105c1ee28fc63477ec55b809
- Sigstore transparency entry: 2337355475
- Sigstore integration time: Aug 3, 2026
Source repository:
- Permalink: latincy/latincy-preprocess@6818d97b179ae2c8e5f771f45d8a96635e77e1fe
- Branch / Tag: refs/tags/v0.4.0
- Owner: https://github.com/latincy
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: release.yml@6818d97b179ae2c8e5f771f45d8a96635e77e1fe
- Trigger Event: push

File details

Details for the file latincy_preprocess-0.4.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.

File metadata

Download URL: latincy_preprocess-0.4.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Upload date: Aug 3, 2026
Size: 497.8 kB
Tags: CPython 3.10+, manylinux: glibc 2.17+ x86-64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for latincy_preprocess-0.4.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Algorithm	Hash digest
SHA256	`090c5fac04c743f9e92ec01a82f08e3fb45d77e0398a9c6727d831a138a602b7`
MD5	`385dc31ba9c55cd11cbc0ce61e20bc6c`
BLAKE2b-256	`595479ff979f133f33016ee024329b2d7d9bb43b84801a8c8721f93fe6281171`

See more details on using hashes here.

Provenance

The following attestation bundles were made for latincy_preprocess-0.4.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl:

Publisher: release.yml on latincy/latincy-preprocess

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: latincy_preprocess-0.4.0-cp310-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
- Subject digest: 090c5fac04c743f9e92ec01a82f08e3fb45d77e0398a9c6727d831a138a602b7
- Sigstore transparency entry: 2337355515
- Sigstore integration time: Aug 3, 2026
Source repository:
- Permalink: latincy/latincy-preprocess@6818d97b179ae2c8e5f771f45d8a96635e77e1fe
- Branch / Tag: refs/tags/v0.4.0
- Owner: https://github.com/latincy
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: release.yml@6818d97b179ae2c8e5f771f45d8a96635e77e1fe
- Trigger Event: push

File details

Details for the file latincy_preprocess-0.4.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.

File metadata

Download URL: latincy_preprocess-0.4.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Upload date: Aug 3, 2026
Size: 488.8 kB
Tags: CPython 3.10+, manylinux: glibc 2.17+ ARM64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for latincy_preprocess-0.4.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
Algorithm	Hash digest
SHA256	`44065907266af07c4cb0d9f68bce1647fb2641d26d4cb7224d4d1ac2fba62ccc`
MD5	`6f39f592196ecacdcd69f8a79743ff18`
BLAKE2b-256	`a2c1e752cdd18eb41583252f93d4f03b477b29d2b078c67d4643cc00a0ec85da`

See more details on using hashes here.

Provenance

The following attestation bundles were made for latincy_preprocess-0.4.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl:

Publisher: release.yml on latincy/latincy-preprocess

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: latincy_preprocess-0.4.0-cp310-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
- Subject digest: 44065907266af07c4cb0d9f68bce1647fb2641d26d4cb7224d4d1ac2fba62ccc
- Sigstore transparency entry: 2337355503
- Sigstore integration time: Aug 3, 2026
Source repository:
- Permalink: latincy/latincy-preprocess@6818d97b179ae2c8e5f771f45d8a96635e77e1fe
- Branch / Tag: refs/tags/v0.4.0
- Owner: https://github.com/latincy
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: release.yml@6818d97b179ae2c8e5f771f45d8a96635e77e1fe
- Trigger Event: push

File details

Details for the file latincy_preprocess-0.4.0-cp310-abi3-macosx_11_0_arm64.whl.

File metadata

Download URL: latincy_preprocess-0.4.0-cp310-abi3-macosx_11_0_arm64.whl
Upload date: Aug 3, 2026
Size: 458.1 kB
Tags: CPython 3.10+, macOS 11.0+ ARM64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for latincy_preprocess-0.4.0-cp310-abi3-macosx_11_0_arm64.whl
Algorithm	Hash digest
SHA256	`238f3754967a7ca38ae9c24d941607316a736671d50509452a34693569f9b12d`
MD5	`b737c1ea3f15d1efce9aff6b7fa51fdf`
BLAKE2b-256	`0173dcd8707249df66dfbb40e20c541e5962e60a571ac169834fcb45b3c306d4`

See more details on using hashes here.

Provenance

The following attestation bundles were made for latincy_preprocess-0.4.0-cp310-abi3-macosx_11_0_arm64.whl:

Publisher: release.yml on latincy/latincy-preprocess

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: latincy_preprocess-0.4.0-cp310-abi3-macosx_11_0_arm64.whl
- Subject digest: 238f3754967a7ca38ae9c24d941607316a736671d50509452a34693569f9b12d
- Sigstore transparency entry: 2337355499
- Sigstore integration time: Aug 3, 2026
Source repository:
- Permalink: latincy/latincy-preprocess@6818d97b179ae2c8e5f771f45d8a96635e77e1fe
- Branch / Tag: refs/tags/v0.4.0
- Owner: https://github.com/latincy
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: release.yml@6818d97b179ae2c8e5f771f45d8a96635e77e1fe
- Trigger Event: push

File details

Details for the file latincy_preprocess-0.4.0-cp310-abi3-macosx_10_12_x86_64.whl.

File metadata

Download URL: latincy_preprocess-0.4.0-cp310-abi3-macosx_10_12_x86_64.whl
Upload date: Aug 3, 2026
Size: 463.1 kB
Tags: CPython 3.10+, macOS 10.12+ x86-64
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/7.0.0 CPython/3.13.14

File hashes

Hashes for latincy_preprocess-0.4.0-cp310-abi3-macosx_10_12_x86_64.whl
Algorithm	Hash digest
SHA256	`2c501c0eb46de2655600b34f4970940f10fe0e6fdfa29ab99310afad4dcee10c`
MD5	`f05df89b4ef3b603e0471e75a26bf3e0`
BLAKE2b-256	`a49f2d21a8daf75ed7533ff947e84b13c2f68ea7cb3c0548df56e44d37f677fb`

See more details on using hashes here.

Provenance

The following attestation bundles were made for latincy_preprocess-0.4.0-cp310-abi3-macosx_10_12_x86_64.whl:

Publisher: release.yml on latincy/latincy-preprocess

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Statement:
- Statement type: https://in-toto.io/Statement/v1
- Predicate type: https://docs.pypi.org/attestations/publish/v1
- Subject name: latincy_preprocess-0.4.0-cp310-abi3-macosx_10_12_x86_64.whl
- Subject digest: 2c501c0eb46de2655600b34f4970940f10fe0e6fdfa29ab99310afad4dcee10c
- Sigstore transparency entry: 2337355486
- Sigstore integration time: Aug 3, 2026
Source repository:
- Permalink: latincy/latincy-preprocess@6818d97b179ae2c8e5f771f45d8a96635e77e1fe
- Branch / Tag: refs/tags/v0.4.0
- Owner: https://github.com/latincy
- Access: public
Publication detail:
- Token Issuer: https://token.actions.githubusercontent.com
- Runner Environment: github-hosted
- Publication workflow: release.yml@6818d97b179ae2c8e5f771f45d8a96635e77e1fe
- Trigger Event: push

Release history Release notifications | RSS feed

0.5.1

Aug 7, 2026 6 files

0.5.0

Aug 7, 2026 6 files

This release

0.4.0 This release

Aug 3, 2026 6 files

0.3.3

Jul 29, 2026 6 files

0.3.2

Jun 30, 2026 15 files

0.3.1

Jun 26, 2026 15 files

Mar 16, 2026 2 files

Mar 1, 2026 2 files

Feb 24, 2026 2 files

Feb 23, 2026 2 files

latincy-preprocess 0.4.0

Installation

Quick Start

Per-Normalizer Usage

U/V Normalization

Long-S OCR Correction

Diacritics and Macrons

Beta Code → Unicode Greek

Ancient Greek Normalization

spaCy Integration

Unified Preprocessor (recommended)

Standalone Components

Rust Backend

Accuracy

U/V Normalization

Long-S Correction

Changelog

Citation

Acknowledgments

License

Project links

Key dates

1 maintainer

Credits

GitHub Statistics

License expression

Requires

Provides Extra

Tags

Classifiers

Download files

Source Distribution

Built Distributions

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

Release history Release notifications | RSS feed