Classes and functions for working with Unicode data.
Python classes and functions for working with Unicode® data. This was initially built with Python 2 in mind but has also been tested with Python 3, PyPy and PyPy3.
This package has the following external dependencies:
- six - for Python 2 to 3 compatibility
Case folding function
Python 3.3 and newer has str.casefold() already built in. This is my attempt at building a case folding function to use with Python 2 and as such was initially only tested with Python 2.7.14. It essentially parses the CaseFolding.txt file that is included in the Unicode® Character Database to build a dictionary that is then used as a lookup table to create a copy of the input string that has been transformed to facilitate caseless comparisons.
A bit more information about how I put this together on my blog.
By default, the casefold(s) function performs full case folding. To use simple case folding, pass the parameter fullcasefold=False (the default is fullcasefold=True). See the comments in CaseFolding.txt for an explanation of the difference between simple and full case folding.
By default, the casefold(s) function will not use the Turkic special case mappings for dotted and dotless ‘i’. To use the Turkic mapping, pass the parameter useturkicmapping=True to the function. See the following web pages for more information on the dotted vs dotless ‘i’:
Using Python 2:
>>> from unicodeutil import casefold >>> s1 = u"weiß" >>> s2 = u"WEISS" >>> casefold(s1) == casefold(s2) True >>> s1 = u"LİMANI" >>> s2 = u"limanı" >>> casefold(s1) == casefold(s2) False >>> casefold(s1, useturkicmapping=True) == casefold(s2, useturkicmapping=True) True
Splitting a Python 2 string into chars, preserving surrogate pairs
The preservesurrogates(s) function will split a string into a list of characters, preserving surrogate pairs.
Using Python 2:
>>> from unicodeutil import preservesurrogates >>> s = u"ABC\U0001e900DeF\U000118a0gHıİ" >>> list(s) [u'A', u'B', u'C', u'\ud83a', u'\udd00', u'D', u'e', u'F', u'\ud806', u'\udca0', u'g', u'H', u'\u0131', u'\u0130'] >>> for c in s: ... print c ... A B C ??? ??? D e F ??? ??? g H ı İ >>> list(preservesurrogates(s)) [u'A', u'B', u'C', u'\U0001e900', u'D', u'e', u'F', u'\U000118a0', u'g', u'H', u'\u0131', u'\u0130'] >>> for c in preservesurrogates(s): ... print(c) ... A B C 𞤀 D e F 𑢠 g H ı İ
Using the latest Unicode® Character Database (UCD)
As of Python 2.7.14, the unicodedata module is still using data from version 5.2.0 of the UCD. The UCD is currently up to version 10.0.0. The UnicodeCharacter namedtuple encapsulates the various properties associated with each Unicode® character, as explained in Unicode Standard Annex #44, UnicodeData.txt. The UnicodeData class represents the contents of the UCD as parsed from the latest UnicodeData.txt found on the Unicode Consortium FTP site. Once an instance of the UnicodeData class has been created, it is possible to do dict style lookups using the Unicode characters, or lookups by name using the lookup_by_name(name) method. The name lookup uses the UAX44-LM2 loose matching rule when doing lookups. Iterating through all of the data is also possible via items(), keys() and values() methods.
Using Python 2:
>>> from unicodeutil import UnicodeData >>> ucd = UnicodeData() >>> ucd[u"ß"] UnicodeCharacter(code=u'U+00DF', name='LATIN SMALL LETTER SHARP S', category='Ll', combining=0, bidi='L', decomposition='', decimal='', digit='', numeric='', mirrored='N', unicode_1_name='', iso_comment='', uppercase='', lowercase='', titlecase='') >>> ucd[u"İ"].name 'LATIN CAPITAL LETTER I WITH DOT ABOVE' >>> ucd.lookup_by_name("latin small letter sharp_s") UnicodeCharacter(code=u'U+00DF', name='LATIN SMALL LETTER SHARP S', category='Ll', combining=0, bidi='L', decomposition='', decimal='', digit='', numeric='', mirrored='N', unicode_1_name='', iso_comment='', uppercase='', lowercase='', titlecase='')
Decomposing Hangul Syllables into Jamo
The function decompose_hangul_syllable(hangul_syllable, fully_decompose=False) takes the Unicode scalar value of a hangul syllable and will either do a canonical decomposition (default, fully_decompose=False) or a full canonical decomposition (fully_decompose=True) of a Hangul syllable. The return value will be a tuple of Unicode scalar values corresponding to the Jamo that the Hangul syllable has been decomposed into. For example (taken from the Unicode Standard, ch. 03, section 3.12, Conjoing Jamo Behavior):
U+D4DB -> <U+D4CC, U+11B6> # Canonical Decomposition (default) U+D4DB -> <U+1111, U+1171, U+11B6> # Full Canonical Decomposition
Using Python 3:
>>> from unicodeutil import decompose_hangul_syllable >>> canonical = decompose_hangul_syllable(0xD4DB) >>> full = decompose_hangul_syllable(0xD4DB, fully_decompose=True) >>> canonical (54476, 4534) >>> ['U+' + hex(jamo)[2:].upper() for jamo in canonical] ['U+D4CC', 'U+11B6'] >>> full (4369, 4465, 4534) >>> ['U+' + hex(jamo)[2:].upper() for jamo in full] ['U+1111', 'U+1171', 'U+11B6']
This is released under an MIT license. See the LICENSE file in this repository for more information.
Release history Release notifications
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size unicodeutil-0.1.dev2-py2.py3-none-any.whl (299.3 kB)||File type Wheel||Python version py2.py3||Upload date||Hashes View|
|Filename, size unicodeutil-0.1.dev2.tar.gz (287.4 kB)||File type Source||Python version None||Upload date||Hashes View|
Hashes for unicodeutil-0.1.dev2-py2.py3-none-any.whl