Skip to main content

Trained Korean Lemmatizer

Project description

한국어 용언 분석기 (Korean Lemmatizer)

한국어의 동사와 형용사의 활용형 (surfacial form) 을 분석합니다. 한국어 용언 분석기는 다음의 기능을 제공합니다.

  1. 입력된 단어를 어간 (stem) 과 어미 (eomi) 으로 분리
  2. 입력된 단어를 원형으로 복원

이 패키지의 구현 원리는 github.io 블로그에 정리하였습니다.

Usage

analyze, lemmatize, conjugate

analyze function returns morphemes of the given predicator word

from soylemma import Lemmatizer

lemmatizer = Lemmatizer()
lemmatizer.analyze('차가우니까')

The return value forms list of tuples because there can be more than one morpheme combination.

[(('차갑', 'Adjective'), ('우니까', 'Eomi'))]

lemmatize function returns lemma of the given predicator word.

lemmatizer.lemmatize('차가우니까')
[('차갑다', 'Adjective')]

If the input word is not predicator such as Noun, it return empty list.

lemmatizer.lemmatize('한국어') # []

conjugate function returns surfacial form. You should put stem and eomi as arguments. It returns all possible surfacial forms for the given stem and eomi.

lemmatizer.conjugate(stem='차갑', eomi='우니까')
lemmatizer.conjugate('예쁘', '었던')
['차가우니까', '차갑우니까']
['예뻤던', '예쁘었던']

update dictionaries and rules

For demonstration, we use dictioanry demo.

어여뻤어 cannot be analyzed because the adjective 어여쁘 does not enrolled in dictionary.

from soylemma import Lemmatizer

lemmatizer = Lemmatizer(dictionary_name='demo')
print(lemmatizer.analyze('어여뻤어')) # []

So, we add the word with tag using add_words function. Do it again. Then you can see the word 어여뻤어 is analyzed.

lemmatizer.add_words('어여쁘', 'Adjective')
lemmatizer.analyze('어여뻤어')
[(('어여쁘', 'Adjective'), ('었어', 'Eomi'))]

However, the word 파랬다 is still not able to be analyzed because the lemmatization rule for surfacial form does not exist.

lemmatizer.analyze('파랬다') # []

So, in this time, we update additional lemmatization rules using add_lemma_rules function.

supplements = {
    '랬': {('랗', '았')}
}

lemmatizer.add_lemma_rules(supplements)

After that, we can see the word 파랬다 is analyzed, and also conjugation of 파랗 + 았다 is available.

lemmatizer.analyze('파랬다')
lemmatizer.conjugate('파랗', '았다')
[(('파랗', 'Adjective'), ('았다', 'Eomi'))]
['파랬다', '파랗았다']

debug on

If you wonder which subwords came up as candidates of (stem, eomi), use debug.

lemmatizer.analyze('파랬다', debug=True)
[DEBUG] word: 파랬다 = 파랗 + 았다, conjugation: 랬 = 랗 + 았
[(('파랗', 'Adjective'), ('았다', 'Eomi'))]

lemmatization rule extractor

You can extract lemmatization rule using extract_rule function.

from soylemma import extract_rule

eojeol = '로드무비였다'
lw = '로드무비이'
lt = 'Adjective'
rw = '었다'
rt = 'Eomi'

extract_rule(eojeol, lw, lt, rw, rt)
('였다', ('이', '었다'))

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for soylemma, version 0.2.0
Filename, size File type Python version Upload date Hashes
Filename, size soylemma-0.2.0-py3.7.egg (135.0 kB) File type Egg Python version 3.7 Upload date Hashes View
Filename, size soylemma-0.2.0-py3-none-any.whl (124.1 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size soylemma-0.2.0.tar.gz (125.2 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page