A library for Korean Jamo split and vectorize.
Project description
A library for Korean Jamo split and vectorize.
Install
pip install jamotools
Unicode of Korean
According to the Version 9.0.0 database of the Unicode Consortium, the blocks specified in Hangul (Korean) in Unicode are as follows.
Hangul Jamo: 1100 ~ 11FF
WON SIGN in Currency Symbols: 20A9
HANGUL DOT TONE MARK in CJK Symbols and Punctuation: 302E ~ 302F
Hangul Compatibility Jamo : 3130 ~ 318F
Hangul in Enclosed CJK Letters and Months: 3200 ~ 321E, 3260 ~ 327F
Hangul Jamo Extended-A : A960 ~ A97F
Hangul Syllables : AC00 ~ D7AF
Hangul Jamo Extended-B : D7B0 ~ D7FF
Halfwidth Hangul variants in Halfwidth and Fullwidth Forms: FFA0 ~ FFDC
FULLWIDTH WON SIGN in Halfwidth and Fullwidth Forms: FFE6
Jamo
Hangul is made of basic letters called Jamo. In unicode, Jamo is defined by several kinds which contain old Hangul that does not use in nowadays. Jamotools only supports modern Hangul Jamo area as follows.
Hangul Jamo: Consist of Choseong, Jungseong, Jongseong. It is divided mordern Hangul and old Hangul that does not use in nowadays. Jamotools supports modern Hangul Jamo area.
1100 ~ 1112 (Choseong)
1161 ~ 1175 (Jungseong)
11A8 ~ 11C2 (Jongseong)
Hangul Compatibility Jamo: It is a Korean Hangul language area that is compatible with the Hangul character standard (KS X 1001). It is not divided Choseong, Jungseong, Jongseong.
3131 ~ 3163 (modern Hangul Jamo area)
Halfwidth Hangul variants: This is the Korean half-width symbol area. Only modern Korean Jamo exists. The general Korean Hangul characterization method is the full-width.
FFA1 ~ FFDC
Manipulating Korean Jamo
API for split syllables and join jamos to syllable is based on hangul-utils.
split_syllables: Converts a string of syllables to a string of jamos, can be select which convert unicode type.
join_jamos: Converts a string of jamos to a string of syllables.
normalize_to_compat_jamo: Normalize a string of jamos to a string of Hangul Compatibility Jamo.
>>> import jamotools
>>> print(jamotools.split_syllable_char(u"안"))
('ㅇ', 'ㅏ', 'ㄴ')
>>> print(jamotools.split_syllables(u"안녕하세요"))
ㅇㅏㄴㄴㅕㅇㅎㅏㅅㅔㅇㅛ
>>> sentence = u"앞 집 팥죽은 붉은 팥 풋팥죽이고, 뒷집 콩죽은 햇콩 단콩 콩죽.우리 집
깨죽은 검은 깨 깨죽인데 사람들은 햇콩 단콩 콩죽 깨죽 죽먹기를 싫어하더라."
>>> s = jamotools.split_syllables(sentence)
>>> print(s)
ㅇㅏㅍ ㅈㅣㅂ ㅍㅏㅌㅈㅜㄱㅇㅡㄴ ㅂㅜㄺㅇㅡㄴ ㅍㅏㅌ ㅍㅜㅅㅍㅏㅌㅈㅜㄱㅇㅣㄱㅗ,
ㄷㅟㅅㅈㅣㅂ ㅋㅗㅇㅈㅜㄱㅇㅡㄴ ㅎㅐㅅㅋㅗㅇ ㄷㅏㄴㅋㅗㅇ ㅋㅗㅇㅈㅜㄱ.ㅇㅜㄹㅣ
ㅈㅣㅂ ㄲㅐㅈㅜㄱㅇㅡㄴ ㄱㅓㅁㅇㅡㄴ ㄲㅐ ㄲㅐㅈㅜㄱㅇㅣㄴㄷㅔ ㅅㅏㄹㅏㅁㄷㅡㄹㅇㅡㄴ
ㅎㅐㅅㅋㅗㅇ ㄷㅏㄴㅋㅗㅇ ㅋㅗㅇㅈㅜㄱ ㄲㅐㅈㅜㄱ ㅈㅜㄱㅁㅓㄱㄱㅣㄹㅡㄹ
ㅅㅣㅀㅇㅓㅎㅏㄷㅓㄹㅏ.
>>> sentence2 = jamotools.join_jamos(s)
>>> print(sentence2)
앞 집 팥죽은 붉은 팥 풋팥죽이고, 뒷집 콩죽은 햇콩 단콩 콩죽.우리 집 깨죽은 검은 깨
깨죽인데 사람들은 햇콩 단콩 콩죽 깨죽 죽먹기를 싫어하더라.
>>> print(sentence == sentence2)
True
Jamotools’ API supports multiple unicode area of Hangul Jamo for manipulating. Also consists of additional API for manipulating Korean jamo.
>>> sentence = u"자모"
>>> jamos1 = jamotools.split_syllables(sentence, jamo_type="JAMO")
>>> print([hex(ord(c)) for c in jamos1])
['0x110C', '0x1161', '0x1106', '0x1169']
>>> sentence1 = jamotools.join_jamos(jamos1)
>>> print(sentence1)
안녕하세요. hello 1
>>> jamos2 = jamotools.split_syllables(sentence, jamo_type="COMPAT")
>>> print([hex(ord(c)) for c in jamos2])
['0x3148', '0x314F', '0x3141', '0x3157']
>>> sentence2 = jamotools.join_jamos(jamos2)
>>> print(sentence2)
안녕하세요. hello 1
>>> jamos3 = jamotools.split_syllables(sentence, jamo_type="HALFWIDTH")
>>> print([hex(ord(c)) for c in jamos3])
['0xFFB8', '0xFFC2', '0xFFB1', '0xFFCC']
>>> sentence3 = jamotools.join_jamos(jamos3)
>>> print(sentence3)
안녕하세요. hello 1
>>> print(sentence == sentence1 == sentence2 == sentence3)
True
>>> normalize1 = jamotools.normalize_to_compat_jamo(jamos1)
>>> normalize2 = jamotools.normalize_to_compat_jamo(jamos2)
>>> normalize3 = jamotools.normalize_to_compat_jamo(jamos3)
>>> print(jamos1 == jamos2 == jamos3)
False
>>> print(normalize1 == normalize2 == normalize3)
True
Vectorize Korean Jamo
Jamotools support vectorize function following RULE. Each RULE is defined how split sentence to Jamo and convert which type of symbols. It can be used character-level Korean text processing.
Vectorizationer: Class for vectorize text by Rule and pad.
>>> v = jamotools.Vectorizationer(rule=jamotools.rules.RULE_1, \
max_length=None, \
prefix_padding_size=0)
>>> print(v.vectorize(u"안녕"))
[13, 21, 45, 4, 27, 62]
Custom RULE
Jamotools can add user’s custom RULE class as following steps.
Make custom RULE class which inherit RuleBase (e.g. Rule2) in rules.py like Rule1.
Add constant for custom RULE like RULE_1.
Modify get_rule function to return custom RULE class.
Then it can be use as same as RULE_1 usage.
>>> v = jamotools.Vectorizationer(rule=jamotools.rules.RULE_2, \
max_length=None, \
prefix_padding_size=0)
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file jamotools-0.1.10.tar.gz
.
File metadata
- Download URL: jamotools-0.1.10.tar.gz
- Upload date:
- Size: 23.9 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | b3dc212ae4fc8dd63ec3ab666c98a6390ac5d28d0adba356ed89360c2d2fb115 |
|
MD5 | 2bc2ca8c3649352cbd5c9d146c62b82a |
|
BLAKE2b-256 | 29bce1cf5a51b739eab1cc157b892052c491642fa54a0d938390340ef3180c91 |
File details
Details for the file jamotools-0.1.10-py2.py3-none-any.whl
.
File metadata
- Download URL: jamotools-0.1.10-py2.py3-none-any.whl
- Upload date:
- Size: 13.6 kB
- Tags: Python 2, Python 3
- Uploaded using Trusted Publishing? No
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2974f7be1a18dfbb392f02a1dfff1900cae3e3fd27d4b56c1d973f148b40ef3f |
|
MD5 | 09c9195d8151a7e4b4372d23c94f6f57 |
|
BLAKE2b-256 | 3dd6ec13c68f7ea6a8085966390d256d183bf8488f8b9770028359acb86df643 |