fyc-normalize 0.1.0

Lightweight value normalizers (emails, phones, US TINs, usernames) usable in any DataFrame or plain Python.

fyc-normalize

See a string, normalize a string. Lightweight value normalizers for cleaning data consistently across projects.

Each data type has one pure function that takes a single value and returns its canonical string form, or None if the value is missing or unparseable. Because the unit of work is a single value, the functions drop into pandas, polars, or plain Python with no special integration. The only core dependency is PyYAML (it loads the bundled email provider profiles); DataFrame and phone-parsing support are optional extras.

Install

pip install fyc-normalize                 # core (PyYAML only)
pip install "fyc-normalize[phone]"        # + phonenumbers, for E.164 phone parsing
pip install "fyc-normalize[domain]"       # + idna, for full IDNA2008 domain Punycode
pip install "fyc-normalize[polars]"       # + polars column helpers
pip install "fyc-normalize[pandas]"       # + pandas column helpers

Usage

Every normalizer is a plain function: pass one value, get back its canonical string form or None. Import what you need:

from fyc_normalize import (
    normalize_email,
    normalize_ani,       # phone numbers (needs the [phone] extra)
    normalize_us_tin,    # SSN / ITIN / ATIN
    normalize_ip,
    normalize_cidr,
    normalize_domain,    # needs the [domain] extra for non-ASCII domains
    normalize_username,
)

normalize_email("[e-mail]")   # "[e-mail]"
normalize_ani("[phone]")              # "[phone]"   (E.164)
normalize_us_tin("123-45-6789")              # "123456789"      (9-digit)
normalize_ip("2001:0DB8::0001")              # "2001:db8::1"    (RFC 5952)
normalize_cidr("192.168.1.5/24")             # "192.168.1.0/24" (host bits masked)
normalize_domain("WWW.Café.COM.")            # "xn--caf-dma.com" (lowercase Punycode)
normalize_username("  Ａdmin ")               # "admin"          (NFKC + lower + trim)

Anything missing or unparseable returns None, so you can filter cleanly:

normalize_email("")            # None
normalize_ip("999.1.1.1")      # None
normalize_cidr("10.0.0.1")     # None  (a /prefix is required; use normalize_ip for bare addresses)

Try it from the Python REPL

$ uv run python

>>> from fyc_normalize import normalize_email, normalize_domain, normalize_cidr
>>> normalize_email("  [e-mail] ")
'[e-mail]'
>>> normalize_domain("_dmarc.Example.com.")
'_dmarc.example.com'
>>> normalize_cidr("2001:db8::1/64")
'2001:db8::/64'

Or run the bundled demo, which exercises every normalizer on messy input:

uv run python examples/demo.py

In a DataFrame

Because each function takes a single value, it drops straight into pandas or polars:

df["email"] = df["email"].map(normalize_email)                       # pandas
df = df.with_columns(pl.col("email").map_elements(normalize_email))  # polars

Or use the optional column helpers (they auto-detect polars vs pandas):

from fyc_normalize.frames import normalize_email_column, normalize_ip_column
df = normalize_email_column(df, "email")     # rewrites the "email" column in place
df = normalize_ip_column(df, "client_ip")

Scope: one job — see a string, normalize a string. Each function takes a single value and returns its canonical form. Nothing more.

Design

src/fyc_normalize/
├── base.py          # BaseNormalizer + blank_to_none() shared pattern
├── emails.py        # EmailNormalizer + normalize_email   (profile-based)
├── anis.py          # AniNormalizer   + normalize_ani     (E.164)
├── us_tins.py       # UsTinNormalizer  + normalize_us_tin  (9-digit SSN/ITIN/ATIN)
├── ips.py           # IpNormalizer/CidrNormalizer + normalize_ip / normalize_cidr
├── domains.py       # DomainNormalizer + normalize_domain  (lowercase Punycode)
├── usernames.py     # UsernameNormalizer + normalize_username (NFKC)
├── frames.py        # optional DataFrame column helpers (lazy polars/pandas)
├── configs/         # email_profiles.yaml — provider profiles (source of truth)
└── rules/           # the actual transformation logic, one class per type

Each type follows the same three-part shape: a Normalizer class, a module-level singleton, and a pure normalize_* function. The substantive logic lives in rules/ so behavior is easy to read, test, and change independently of the wrappers.

Adding a new normalizer

1. Add a rule class in rules/<type>_rules.py.
2. Add <type>s.py with a Normalizer subclass, singleton, and normalize_<type> function.
3. Export it in __init__.py and (optionally) add a column helper in frames.py.

Email profiles

Normalization rules are selected per provider by domain, defined in configs/email_profiles.yaml (loaded automatically). A provider with no domains is the generic fallback. Each rule name must exist on EmailNormalizingRules and be registered in emails.RULE_REGISTRY. To customize, edit the bundled YAML or point at your own copy:

from fyc_normalize.emails import EmailNormalizer
norm = EmailNormalizer(config_path="my_profiles.yaml")
norm.apply("[e-mail]")   # "[e-mail]"

Currently configured providers: gmail, protonmail, yahoo, fastmail (with its full alias-domain list), and a generic fallback for everything else.

Develop & test (uv)

This project uses uv. It manages the virtual environment, the Python version, and dependency installation for you — there is no venv to create or activate by hand.

Install uv if you don't have it (curl -LsSf https://astral.sh/uv/install.sh | sh on macOS/Linux), then from the project root:

uv sync                 # create .venv + install the package and the dev tooling
uv run pytest           # run the test suite — expect "40 passed"

uv sync installs fyc-normalize itself (editable) plus the dev dependency group (pytest, phonenumbers, idna). uv run executes the command inside that managed environment, so you never source .../activate yourself.

Common commands:

uv sync --all-extras            # also install polars + pandas (for the frame helpers)
uv run pytest -q                # quiet test run
uv run python examples/demo.py  # see every normalizer on sample input
uv run python                   # open a REPL with the package importable
uv pip list | grep fyc          # confirm the project installed: fyc-normalize 0.1.0 (editable)

Troubleshooting — if uv run pytest reports ModuleNotFoundError: No module named 'fyc_normalize', you almost certainly have an old, manually-activated virtual environment shadowing uv's. Reset it:

deactivate            # leave any activated venv (ignore "command not found")
rm -rf .venv          # drop the stale environment
uv sync               # uv recreates .venv AND installs the project
uv run pytest

With uv you don't activate anything — always go through uv run.

(Plain pip install -e ".[dev]" still works if you'd rather not use uv.)

Every push and pull request runs the test suite across Python 3.10–3.13 via GitHub Actions (.github/workflows/ci.yml).

Releasing

Publishing is automated through GitHub Actions using PyPI Trusted Publishing (OIDC) — no API tokens or stored secrets.

Two separate things to keep straight:

• The publish workflow runs on every merge to main (i.e. any develop → main PR). It builds the sdist + wheel, runs twine check, and calls the upload step every time.
• A new version is only released when you bump version in pyproject.toml. The upload uses skip-existing, so if the version already exists on PyPI (you merged without changing it), the upload is skipped and nothing else happens — the job still succeeds. If the version is new, that version is uploaded to PyPI and the workflow automatically creates the matching v<version> git tag and a GitHub Release, with notes pulled from the matching section of CHANGELOG.md.

In short: merging the PR is what runs the job; bumping the version number is what makes an actual release (PyPI + tag + GitHub Release) go out. You don't create tags or releases by hand — the workflow does it for you.

One-time setup (before the first publish):

1. On PyPI, go to your account → Publishing → Add a pending publisher, and register:
   • PyPI Project Name: fyc-normalize
   • Owner / Repository: your GitHub org/repo
   • Workflow name: publish.yml
   • Environment name: pypi
2. In the GitHub repo, create an environment named pypi (Settings → Environments → New environment). Optionally add a required reviewer so each publish needs approval.
3. Protect main (Settings → Branches) so changes land only via pull request.
4. Ensure Actions can create releases: Settings → Actions → General → Workflow permissions should allow read and write (the workflow also requests contents: write explicitly for the release job).

To ship a new version (work on develop, release via PR):

1. Bump version in pyproject.toml. This is the one step that makes a new release happen — if you skip it, merging still runs the workflow but publishes nothing new. (pyproject.toml is the single source of truth; fyc_normalize.__version__ reads it from the installed package metadata.)
2. Move the [Unreleased] notes into a new version section in CHANGELOG.md.
3. Open a PR from develop to main; CI runs the test matrix on it.
4. Merge the PR. The push to main runs publish.yml, which uploads the new version to PyPI and creates the v<version> tag + GitHub Release automatically.

Merges that don't touch the version (docs, refactors, etc.) are fine — the workflow runs, skips the upload, and skips the release because the tag already exists. No manual tagging needed either way.

To dry-run the build without publishing, trigger the workflow manually (Actions → Publish to PyPI → Run workflow); the publish job is skipped on manual runs.

Note: normalized US TINs (SSNs, ITINs, ...) are sensitive plaintext PII. Treat any table that stores them as sensitive.