text-iron

Dependency-free Persian/multilingual text normalization for search and matching (queries, titles, body content). One implementation, pinned by NORM_VERSION, so text normalized at different times stays byte-identical — drift silently breaks matching.

Use

from text_iron import normalize_query, normalize_title, normalize_content

normalize_query("Café كيك ۱۲۳")        # -> "cafe کیک 123"
normalize_title("کتاب: جلد ۲")          # -> "کتاب: جلد 2"
normalize_content("بند اول\n\nبند دوم")  # -> keeps the paragraph break

The output is a matching / embedding key, not display text. Always keep the original around for display and snippets.

URLs

text_iron.url is a URL normalizer for whatever keys URLs — a crawler's seen-set, an index, a store of search results — so that everything meets on one key:

from text_iron.url import normalize_url, URL_NORMALIZATION_VERSION

normalize_url("HTTP://Example.IR:80/a/./b?utm_source=x&b=2&a=1#top").value
# -> "http://example.ir/a/b?a=1&b=2"

It is not text normalization and uses none of it: a URL is an identifier, and folding its letters or digits would change it. Its rules are versioned by URL_NORMALIZATION_VERSION, separately from NORM_VERSION.

Pipelines

All three share one folding core and are idempotent.

Components

Composable, single-purpose, pure: to_unicode_form, fold_persian_letters, strip_arabic_diacritics, fold_latin_diacritics, normalize_digits, to_lowercase, canonicalize_zwnj, filter_charset (via CharPolicy), flatten_whitespace, normalize_content_whitespace, fingerprint. Build a new pipeline by composing these, then pin a new NORM_VERSION.

Design decisions worth knowing

• Homophones (س/ص/ث, ز/ذ/ض/ظ, …) are NOT folded — they are distinct letters and real spelling signal.
• Hamza forms (أ/إ/ؤ/ئ), ة, ۀ are NOT folded — conservative default; folding raises recall. Flip in fold_persian_letters / the table if desired.
• NFKC is used, which rewrites m²→m2, ½→1⁄2, ①→1. Switch a pipeline's _fold(..., unicode_form="NFC") if a use case must keep those glyphs.
• Query drops non-Persian/Latin scripts; content keeps every script. The charset policy is the main knob (CharPolicy).

Install

pip install text-iron

Test

uv run pytest tests            # or:
python tests/test_text_iron.py