Skip to main content

Unicode normalization forms (NFC, NFKC, NFD, NFKD). A library independent from the Python core Unicode database. This package supports version 14.0 of the Unicode Standard.

Project description

pyunormalize

PyPI Version PyPI License

A pure Python implementation of the Unicode Normalization Algorithm independent from the Python core Unicode database. This package supports version 14.0 of the Unicode Standard (released September 14, 2021). It has been successfully tested against the Unicode test file.

To get the version of the Unicode character database currently used:

>>> from pyunormalize import UCD_VERSION
>>> UCD_VERSION
'14.0.0'

For the formal specification of the Unicode Normalization Algorithm, see Section 3.11, Normalization Forms, in the Unicode core specification.

Installation

pip install pyunormalize

Example usage

>>> from pyunormalize import NFC, NFD, NFKC, NFKD
>>> s = "élève"  # "\u00E9\u006C\u00E8\u0076\u0065"
>>> nfc = NFC(s)
>>> nfd = NFD(s)
>>> nfc == s
True
>>> nfd == nfc
False
>>> " ".join([f"{ord(x):04X}" for x in nfc])
'00E9 006C 00E8 0076 0065'
>>> " ".join([f"{ord(x):04X}" for x in nfd])
'0065 0301 006C 0065 0300 0076 0065'
>>> 
>>> s = "⑴ ffi ²"
>>> NFC(s), NFKC(s), NFD(s), NFKD(s)
('⑴ ffi ²', '(1) ffi 2', '⑴ ffi ²', '(1) ffi 2')

>>> from pyunormalize import normalize
>>> forms = ["NFC", "NFD", "NFKC", "NFKD"]
>>> [normalize(f, "\u017F\u0307\u0323") for f in forms]
['ẛ̣', 'ẛ̣', 'ṩ', 'ṩ']

References

License

The pyunormalize library is released under an MIT license.

The Unicode Standard files this library makes use of are licensed under the Unicode License Agreement for Data Files and Software. Please consult the UNICODE, INC. LICENSE AGREEMENT prior to use.

Changes

CHANGELOG

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pyunormalize-14.0.0.post2.tar.gz (82.7 kB view details)

Uploaded Source

File details

Details for the file pyunormalize-14.0.0.post2.tar.gz.

File metadata

  • Download URL: pyunormalize-14.0.0.post2.tar.gz
  • Upload date:
  • Size: 82.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.60.0 CPython/3.8.5

File hashes

Hashes for pyunormalize-14.0.0.post2.tar.gz
Algorithm Hash digest
SHA256 9fdca0943d8145bcb94d84259258bc593874a93ee85aedb6896c13225ac0726c
MD5 04dc797e054e009b54d8444900203cbe
BLAKE2b-256 2b8adde99d2740cdef576780a205f8b6ba534c430ce7b9ef6975acca945271ef

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page