Unicode normalization forms (NFC, NFKC, NFD, NFKD). A library independent from the Python core Unicode database. This package supports version 14.0 of the Unicode Standard.
Project description
pyunormalize
A pure Python implementation of the Unicode Normalization Algorithm independent from the Python core Unicode database. This package supports version 14.0 of the Unicode Standard (released September 14, 2021). It has been successfully tested against the Unicode test file.
To get the version of the Unicode character database currently used:
>>> from pyunormalize import UCD_VERSION
>>> UCD_VERSION
'14.0.0'
For the formal specification of the Unicode Normalization Algorithm, see Section 3.11, Normalization Forms, in the Unicode core specification.
Installation
pip install pyunormalize
Example usage
>>> from pyunormalize import NFC, NFD, NFKC, NFKD
>>> s = "élève" # "\u00E9\u006C\u00E8\u0076\u0065"
>>> nfc = NFC(s)
>>> nfd = NFD(s)
>>> nfc == s
True
>>> nfd == nfc
False
>>> " ".join([f"{ord(x):04X}" for x in nfc])
'00E9 006C 00E8 0076 0065'
>>> " ".join([f"{ord(x):04X}" for x in nfd])
'0065 0301 006C 0065 0300 0076 0065'
>>>
>>> s = "⑴ ffi ²"
>>> NFC(s), NFKC(s), NFD(s), NFKD(s)
('⑴ ffi ²', '(1) ffi 2', '⑴ ffi ²', '(1) ffi 2')
>>> from pyunormalize import normalize
>>> forms = ["NFC", "NFD", "NFKC", "NFKD"]
>>> [normalize(f, "\u017F\u0307\u0323") for f in forms]
['ẛ̣', 'ẛ̣', 'ṩ', 'ṩ']
References
- https://www.unicode.org/versions/Unicode14.0.0/ch03.pdf#G49537
- https://unicode.org/reports/tr15/
- https://www.unicode.org/Public/14.0.0/ucd/CompositionExclusions.txt
- https://www.unicode.org/Public/14.0.0/ucd/DerivedNormalizationProps.txt
- https://www.unicode.org/Public/14.0.0/ucd/UnicodeData.txt
- https://www.unicode.org/Public/14.0.0/ucd/NormalizationTest.txt
License
The pyunormalize library is released under an MIT license.
The Unicode Standard files this library makes use of are licensed under the Unicode License Agreement for Data Files and Software. Please consult the UNICODE, INC. LICENSE AGREEMENT prior to use.
Changes
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
File details
Details for the file pyunormalize-14.0.0.post2.tar.gz
.
File metadata
- Download URL: pyunormalize-14.0.0.post2.tar.gz
- Upload date:
- Size: 82.7 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.4.2 importlib_metadata/4.8.1 pkginfo/1.7.1 requests/2.25.1 requests-toolbelt/0.9.1 tqdm/4.60.0 CPython/3.8.5
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9fdca0943d8145bcb94d84259258bc593874a93ee85aedb6896c13225ac0726c |
|
MD5 | 04dc797e054e009b54d8444900203cbe |
|
BLAKE2b-256 | 2b8adde99d2740cdef576780a205f8b6ba534c430ce7b9ef6975acca945271ef |