KoEDA

Easy Data Augmentation for Korean

This is a project that re-implemented Easy data augmentation and A Easier Data Augmentation, which were implemented for English, to fit Korean.

Prerequisites

• python >= 3.7

Installation

This repository is tested on Python 3.7 - 3.9.

KoEDA can be installed using pip as follows:

$ pip install koeda

Quick Start

• EDA

from koeda import EDA


eda = EDA(
    morpheme_analyzer="Okt", alpha_sr=0.3, alpha_ri=0.3, alpha_rs=0.3, prob_rd=0.3
)

text = "아버지가 방에 들어가신다"

result = eda(text)
print(result)
# 아버지가 정실에 들어가신다

result = eda(text, p=(0.9, 0.9, 0.9, 0.9), repetition=2)
print(result)
# ['아버지가 객실 아빠 안방 방에 정실 들어가신다', '아버지가 탈의실 방 휴게실 에 안방 탈의실 들어가신다']

• AEDA

from koeda import AEDA


aeda = AEDA(
    morpheme_analyzer="Okt", punc_ratio=0.3, punctuations=[".", ",", "!", "?", ";", ":"]
)

text = "어머니가 집을 나가신다"

result = aeda(text)
print(result)
# 어머니가 ! 집을 , 나가신다

result = aeda(text, p=0.9, repetition=2)
print(result)
# ['! 어머니가 ! 집 ; 을 ? 나가신다', '. 어머니 ? 가 . 집 , 을 , 나가신다']

Augmenters

• EasyDataAugmentation (EDA)
• AEasierDataAugmentation (AEDA)
• RandomDeletion (RD)
• RandomInsertion (RI)
• SynonymReplacement (SR)
• RandomSwap (RS)

There are two ways to load Augmenter.

The first is to use the full name.

from koeda import EasyDataAugmentation

The second is to use abbreviations.

from koeda import EDA

Usage

• EDA

augmenter = EDA(
              morpheme_analyzer: str = None,  # Default = "Okt"
              alpha_sr: float = 0.1,
              alpha_ri: float = 0.1,
              alpha_rs: float = 0.1,
              prob_rd: float = 0.1
            )

result = augmenter(
            data: Union[List[str], str], 
            p: List[float] = None,  # Default = (0.1, 0.1, 0.1, 0.1)
            repetition: int = 1
          )

• AEDA

augmenter = AEDA(
              morpheme_analyzer: str = None,  # Default = "Okt"
              punc_ratio: float = 0.3,
              punctuations: List[str] = None  # default = ('.', ',', '!', '?', ';', ':')
            )

result = augmenter(
            data: Union[List[str], str], 
            p: float = None,  # Default = 0.3 
            repetition: int = 1
          )

• The others (RD, RI, SR, RS)

augmenter = RD(
              morpheme_analyzer: str = None, 
            )

augmenter = RI(
              morpheme_analyzer: str = None, 
              stopword: bool = False
            )

augmenter = SR(
              morpheme_analyzer: str = None, 
              stopword: bool = False
            )

augmenter = RS(
              morpheme_analyzer: str = None, 
            )

result = augmenter(
            data: Union[List[str], str], 
            p: float = 0.1,
            repetition: int = 1
          )

Reference

Easy Data Augmentation Paper
Easy Data Augmentation Repository
A Easier Data Augmentation Paper
A Easier Data Augmentation Repository
Korean WordNet