Skip to main content

An implementation of the Unicode algorithm for breaking code point sequences into extended grapheme clusters as specified in UAX #29

Project description

pyuegc

An implementation of the Unicode algorithm for breaking strings of text (i.e., code point sequences) into extended grapheme clusters (“user-perceived characters”) as specified in UAX #29, “Unicode Text Segmentation”. This package supports version 15.0 of the Unicode standard (released on September 13, 2022). It has been thoroughly tested against the Unicode test file.

Installation

The easiest method to install is using pip:

pip install pyuegc

UCD version

To get the version of the Unicode character database currently used:

>>> from pyuegc import UCD_VERSION
>>> UCD_VERSION
'15.0.0'

Example usage

from pyuegc import EGC

for s in ["e\u0301le\u0300ve", "Z̷̳̎a̸̛ͅl̷̻̇g̵͉̉o̸̰͒", "기운찰만하다"]:
    egc = EGC(s)
    print(f"{len(s):>2}, {len(egc)}: {egc}")

# Output
#  7, 5: ['é', 'l', 'è', 'v', 'e']
# 20, 5: ['Z̷̳̎', 'a̸̛ͅ', 'l̷̻̇', 'g̵͉̉', 'o̸̰͒']
# 15, 6: ['기', '운', '찰', '만', '하', '다']


s = "ai\u0302ne\u0301e"  # aînée
print("".join(reversed(s)))
print("".join(reversed(EGC(s))))

# Output
# éen̂ia -> wrong (diacritics are messed up)
# eénîa -> right (regardless of the Unicode normalization form)

Related resources

This implementation is based on the following resources:

Licenses

The code is available under the MIT license.

Usage of Unicode data files is governed by the UNICODE TERMS OF USE, a copy of which is included as UNICODE-LICENSE.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pyuegc-15.0.0.tar.gz (66.8 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page