latincy-preprocess

LatinCy Preprocess

Latin text preprocessing: U/V normalization, long-s OCR correction, diacritics stripping, macron removal, and Beta Code → Unicode Greek conversion — with optional Rust acceleration and spaCy integration.

Consolidates latincy-uv and latincy-long-s into a single package.

Installation

pip install latincy-preprocess

For spaCy pipeline components:

pip install latincy-preprocess[spacy]

Quick Start

from latincy_preprocess import normalize

normalize("Gallia eft omnis diuisa in partes tres")
# 'Gallia est omnis divisa in partes tres'

Per-Normalizer Usage

U/V Normalization

Converts u-only Latin spelling to proper u/v distinction using rule-based analysis:

from latincy_preprocess import normalize_uv

normalize_uv("Arma uirumque cano")
# 'Arma virumque cano'

Rules handle digraphs (qu), trigraphs (ngu), morphological exceptions (cui, fuit), positional context (initial, intervocalic, post-consonant), and case preservation.

Long-S OCR Correction

Corrects OCR errors where historical long-s (ſ) was misread as f, using n-gram frequency analysis from Latin treebank data:

from latincy_preprocess import LongSNormalizer

normalizer = LongSNormalizer()

word, rules = normalizer.normalize_word_full("ftatua")
# ('statua', [TransformationRule(...)])

text = normalizer.normalize_text_full("funt in fundamento reipublicae ftatua")
# 'sunt in fundamento reipublicae statua'

Two-pass strategy: Pass 1 applies high-confidence rules (impossible bigrams like ft, fp, fc). Pass 2 uses 4-gram frequency disambiguation for ambiguous word-initial f- patterns.

Diacritics and Macrons

from latincy_preprocess import strip_diacritics, strip_macrons

strip_macrons("ārma")
# 'arma'

strip_diacritics("λόγος")
# 'λογος'

Beta Code → Unicode Greek

Latin prose corpora often encode embedded Greek quotations as TLG/Perseus-style Beta Code. Convert it to polytonic Unicode (NFC):

from latincy_preprocess import beta_to_unicode

beta_to_unicode("zei/dwros a)/roura")
# 'ζείδωρος ἄρουρα'

Note: this transliterates every ASCII letter to Greek, so apply it only to isolated Beta Code spans, not mixed Latin/Greek text. Use is_betacode() to guard or segment input:

from latincy_preprocess import beta_to_unicode, is_betacode

span = "a)/nqrwpos"
clean = beta_to_unicode(span) if is_betacode(span) else span
# 'ἄνθρωπος'  —  Latin spans are left untouched

is_betacode() is a heuristic (Beta Code written with no diacritics is indistinguishable from Latin), but it reliably catches accented Greek and ignores ordinary Latin punctuation.

spaCy Integration

Three pipeline components are available as spaCy factories:

Unified Preprocessor (recommended)

Chains long-s correction → U/V normalization in the correct order:

import spacy

nlp = spacy.blank("la")
nlp.add_pipe("latin_preprocessor")

doc = nlp("Gallia eft omnis diuisa in partes tres")
doc._.preprocessed          # 'Gallia est omnis divisa in partes tres'
doc[2]._.preprocessed       # 'est'
doc[2]._.preprocessed_lemma # normalized lemma

Either normalizer can be disabled:

nlp.add_pipe("latin_preprocessor", config={"uv": False})
nlp.add_pipe("latin_preprocessor", config={"long_s": False})

Standalone Components

nlp.add_pipe("uv_normalizer")
# doc._.uv_normalized, token._.uv_normalized, token._.uv_normalized_lemma

nlp.add_pipe("long_s_normalizer")
# doc._.long_s_normalized, token._.long_s_normalized

Rust Backend

When compiled with maturin, a Rust backend provides ~3x throughput for both normalizers. The backend is selected automatically:

from latincy_preprocess import backend

backend()  # 'rust' or 'python'

The Python backend is fully functional and used as the fallback.

Accuracy

U/V Normalization

Dataset	Accuracy
Curated test set (100 sentences)	100%
UD Latin PROIEL (~21K u/v chars)	~98%
UD Latin Perseus (~18K u/v chars)	~97%

Long-S Correction

Pass 1 rules have a 0.00% false positive rate. Pass 2 disambiguation uses a protected allowlist of ~170 common Latin f- words (inline in long_s/_rules.py) plus n-gram frequency tables (JSON files in long_s/data/ngrams/).

Changelog

See CHANGELOG.md for release history.

Citation

@software{latincy_preprocess,
  title = {latincy-preprocess: Text Preprocessing for LatinCy Projects},
  author = {Burns, Patrick J.},
  year = {2026},
  url = {https://github.com/latincy/latincy-preprocess}
}

Acknowledgments

The betacode submodule adapts the Beta Code → Unicode conversion tables and algorithm from the Classical Language Toolkit (cltk.alphabet.grc.beta_to_unicode), used under the MIT License (Copyright © 2013 Classical Language Toolkit). It is reimplemented here on the Python standard library so the package remains dependency-free.

License

MIT

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

latincy_preprocess-0.3.2.tar.gz (165.6 kB view details)

Uploaded Jun 30, 2026 Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

The dropdown lists show the available interpreters, ABIs, and platforms. Enable javascript to be able to filter the list of wheel files.

latincy_preprocess-0.3.2-pp311-pypy311_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (491.8 kB view details)

Uploaded Jun 30, 2026 PyPymanylinux: glibc 2.17+ x86-64

latincy_preprocess-0.3.2-pp311-pypy311_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (484.9 kB view details)

Uploaded Jun 30, 2026 PyPymanylinux: glibc 2.17+ ARM64

latincy_preprocess-0.3.2-pp310-pypy310_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (485.0 kB view details)

Uploaded Jun 30, 2026 PyPymanylinux: glibc 2.17+ ARM64

latincy_preprocess-0.3.2-cp313-cp313-win_amd64.whl (354.0 kB view details)

Uploaded Jun 30, 2026 CPython 3.13Windows x86-64

latincy_preprocess-0.3.2-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (490.4 kB view details)

Uploaded Jun 30, 2026 CPython 3.13manylinux: glibc 2.17+ x86-64

latincy_preprocess-0.3.2-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (482.9 kB view details)

Uploaded Jun 30, 2026 CPython 3.13manylinux: glibc 2.17+ ARM64

latincy_preprocess-0.3.2-cp313-cp313-macosx_11_0_arm64.whl (445.2 kB view details)

Uploaded Jun 30, 2026 CPython 3.13macOS 11.0+ ARM64

latincy_preprocess-0.3.2-cp313-cp313-macosx_10_12_x86_64.whl (451.0 kB view details)

Uploaded Jun 30, 2026 CPython 3.13macOS 10.12+ x86-64

latincy_preprocess-0.3.2-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (490.8 kB view details)

Uploaded Jun 30, 2026 CPython 3.12manylinux: glibc 2.17+ x86-64

latincy_preprocess-0.3.2-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (483.2 kB view details)

Uploaded Jun 30, 2026 CPython 3.12manylinux: glibc 2.17+ ARM64

latincy_preprocess-0.3.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (490.8 kB view details)

Uploaded Jun 30, 2026 CPython 3.11manylinux: glibc 2.17+ x86-64

latincy_preprocess-0.3.2-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (483.8 kB view details)

Uploaded Jun 30, 2026 CPython 3.11manylinux: glibc 2.17+ ARM64

latincy_preprocess-0.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (491.0 kB view details)

Uploaded Jun 30, 2026 CPython 3.10manylinux: glibc 2.17+ x86-64

latincy_preprocess-0.3.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (483.9 kB view details)

Uploaded Jun 30, 2026 CPython 3.10manylinux: glibc 2.17+ ARM64

File details

Details for the file latincy_preprocess-0.3.2.tar.gz.

File metadata

Download URL: latincy_preprocess-0.3.2.tar.gz
Upload date: Jun 30, 2026
Size: 165.6 kB
Tags: Source
Uploaded using Trusted Publishing? Yes
Uploaded via: twine/6.1.0 CPython/3.13.12

File hashes

Hashes for latincy_preprocess-0.3.2.tar.gz
Algorithm	Hash digest
SHA256	`750d5dd50da11666bd48cf41099eb81377d00dbda7e47ada8b23943f0855579f`
MD5	`b6dca8591a9b3b7febc0f76e4bcbd961`
BLAKE2b-256	`3ce1c18b925a73c4ddff542db3e7a487218659e935b6b002e45ce6356242e178`

Algorithm	Hash digest
SHA256	`2c95522f422bd54d6244fbc70af542ac7c5235fbb262821783845659347db734`
MD5	`039f932c211e06816665b8bda32646da`
BLAKE2b-256	`6607dede72bada5b401f80e53d2184d14d89a4308c6e19a0a0e80bdb3b2852c9`

Algorithm	Hash digest
SHA256	`cd021d10c7df16d2cc47e50942098c2d281d3211f6e3b51168180c1c0dcd08e5`
MD5	`3c241b0680eb83f807cde6e387f3a3a5`
BLAKE2b-256	`e680b5f85d4bdedd108b776848121fb2e46f959950f1efea17f02672de3e8895`

Algorithm	Hash digest
SHA256	`d0ae25d8c1ab121dcac25903534c36fc44928e6a0b1d91c3340e58e9ceb248a4`
MD5	`e89fd3e4cdb32adb715ff27d9c077405`
BLAKE2b-256	`191b17e7bfccca11f619e81d1cd95ef77a87a45aeaa9973da824bf6ee93f8a60`

Algorithm	Hash digest
SHA256	`b4bb6ecd39083221f48c2806dfd324f36a8b5c20ddb47b08477cd79ccf2e614a`
MD5	`6b1e6b0256c9b90b237ed3c5c03debdd`
BLAKE2b-256	`015079c2e501f1c9151fc0a96074549d0a8c827e4286303f5a03af13de407466`

Algorithm	Hash digest
SHA256	`8e27de675fa079088cbcd652099ac60de6dfdf00a3474300e11954a0adab1595`
MD5	`fdb9c260fabe47679a4f446fd6ce9112`
BLAKE2b-256	`70f4d91290b06a514a501f081022b3949cb73c9c6d1d466bca796305ec347f5e`

Algorithm	Hash digest
SHA256	`a08d707a051fc6dfd36d3062de51d6c54530aa9644860e61cf7fcd8b96787286`
MD5	`9bc85843210194d9f807fa0c8b293136`
BLAKE2b-256	`ea63dbdbcfbd65f2f6be1960b8d590dcc856642a5fe25369daa6573f3e63a619`

Algorithm	Hash digest
SHA256	`eff25feb3d2032b607a4cdbcf4c7207855d1a68b77e865cb34c44cd4cc73a869`
MD5	`5ac4b2734eb5958a2b474fed0b9dbea7`
BLAKE2b-256	`4957e5d3c6ffb724cc58ca99273b4e7a150e768230d25f706116c98edf6a4e05`

Algorithm	Hash digest
SHA256	`ed5f1f9a889400644f6cfe704bbac5e0a4ae55ebacd47ca39533410457fae401`
MD5	`abf5799a5df17421d6cae89a26a98038`
BLAKE2b-256	`ab3e6372b3d9d3e29e91ce946934257632c89be4026a3874feead4a874f6ff7a`

Algorithm	Hash digest
SHA256	`27c4d78c18c73d548eaaa12149173ef41b520c99af7af0d109872dde33bdad64`
MD5	`a9ed5407e03e804349b297c458a97152`
BLAKE2b-256	`77d638fe09dc3e711febaa2204c75e8ce652c12ada1118333448e1fad7ed230b`

Algorithm	Hash digest
SHA256	`88d014d42ea1ba9388b02171f821b78060915f6d27855166b7b1e04933f61bcc`
MD5	`82138832c045565a6bf398562fd0daa4`
BLAKE2b-256	`d71b56afb4427a52fc7b2df0169b521eebade9785380579b14c63aa6f3b9d87a`

Algorithm	Hash digest
SHA256	`d4d1575ce921be0e5c0c554b8036f5463550efb22725f81529c90429c10adbc5`
MD5	`62af1ec000b0b13660e5bea37a0400b0`
BLAKE2b-256	`7fd5e8f68a634cc60277b5535d330029a410251fed14e35ead941a4b2dda6766`

Algorithm	Hash digest
SHA256	`236501a66c9f8d089be23b45fada8d9b1351a6de629871aa43dc74966040590c`
MD5	`8b545e231afb97ecf45e23f92af4c921`
BLAKE2b-256	`26bedd8b9de7d9af5ab5523bdda7dc9b345717919295ddfa22392c7af0a8c36f`

Algorithm	Hash digest
SHA256	`3c72f35c5ce2eb00d82192de4283035624ecc4c805c8a2fae0e0a433e1cf0a6a`
MD5	`e5e9fda6a781a95cc22c5eafe78cce1b`
BLAKE2b-256	`691c90a42cd43b9bd6a5b1debdfdcd261effd37fda52f629cf8dd959852ec412`

Algorithm	Hash digest
SHA256	`c452bad478c7d345a1809e871bfdcf31a047a3d51469de8573103b3da78b5b62`
MD5	`29738b4a373591ccd01da6fe3b7b93dd`
BLAKE2b-256	`f53e543122f56dc4e568bf17898c63cbf1b40e3bdbc1048326ff197a77e17e3b`

latincy-preprocess 0.3.2

Installation

Quick Start

Per-Normalizer Usage

U/V Normalization

Long-S OCR Correction

Diacritics and Macrons

Beta Code → Unicode Greek

spaCy Integration

Unified Preprocessor (recommended)

Standalone Components

Rust Backend

Accuracy

U/V Normalization

Long-S Correction

Changelog

Citation

Acknowledgments

License

Project links

Key dates

1 maintainer

Credits

GitHub Statistics

License expression

Requires

Provides Extra

Tags

Classifiers

Download files

Source Distribution

Built Distributions

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance

File details

File metadata

File hashes

Provenance