kumparan's NLP Services

nlp-id is a collection of modules which provides various functions for Natural Language Processing for Bahasa Indonesia. This repository contains all source code related to NLP services.

Installation

To install nlp-id, use the following command:

$ pip install nlp-id     

Usage

Description on how to use the lemmatizer, tokenizer, POS-tagger, etc. will be explained in more detail in this section.

Lemmatizer

Lemmatizer is used to get the root words from every word in a sentence.

from nlp_id.lemmatizer import Lemmatizer 
lemmatizer = Lemmatizer() 
lemmatizer.lemmatize('Saya sedang mencoba') 
# saya sedang coba 

Tokenizer

Tokenizer is used to convert text into tokens of word, punctuation, number, date, email, URL, etc. There are two kinds of tokenizer in this repository, standard tokenizer and phrase tokenizer. The standard tokenizer tokenizes the text into separate tokens where the word tokens are single-word tokens. Tokens that started with ku- or ended with -ku, -mu, -nya, -lah, -kah will be split if it is personal pronoun or particle.

from nlp_id.tokenizer import Tokenizer 
tokenizer = Tokenizer() 
tokenizer.tokenize('Lionel Messi pergi ke pasar di daerah Jakarta Pusat.') 
# ['Lionel', 'Messi', 'pergi', 'ke', 'pasar', 'di', 'daerah', 'Jakarta', 'Pusat', '.']

tokenizer.tokenize('Lionel Messi pergi ke rumahmu di daerah Jakarta Pusat.') 
# ['Lionel', 'Messi', 'pergi', 'ke', 'rumah', 'mu', 'di', 'daerah', 'Jakarta', 'Pusat', '.']

The phrase tokenizer tokenizes the text into separate tokens where the word tokens are phrases (single or multi-word tokens).

from nlp_id.tokenizer import PhraseTokenizer 
tokenizer = PhraseTokenizer() 
tokenizer.tokenize('Lionel Messi pergi ke pasar di daerah Jakarta Pusat.') 
# ['Lionel Messi', 'pergi', 'ke', 'pasar', 'di', 'daerah', 'Jakarta Pusat', '.']

POS Tagger

POS tagger is used to obtain the Part-Of-Speech tag from a text. There are two kinds of POS tagger in this repository, standard POS tagger and phrase POS tagger. The tokens in standard POS Tagger are single-word tokens, while the tokens in phrase POS Tagger are phrases (single or multi-word tokens).

from nlp_id.postag import PosTag
postagger = PosTag() 
postagger.get_pos_tag('Lionel Messi pergi ke pasar di daerah Jakarta Pusat.') 
# [('Lionel', 'NNP'), ('Messi', 'NNP'), ('pergi', 'VB'), ('ke', 'IN'), ('pasar', 'NN'), ('di', 'IN'), ('daerah', 'NN'),  
  ('Jakarta', 'NNP'), ('Pusat', 'NNP'), ('.', 'SYM')]

postagger.get_phrase_tag('Lionel Messi pergi ke pasar di daerah Jakarta Pusat.') 
# [('Lionel Messi', 'NP'), ('pergi', 'VP'), ('ke', 'IN'), ('pasar', 'NN'), ('di', 'IN'), ('daerah', 'NN'), 
  ('Jakarta Pusat', 'NP'), ('.', 'SYM')]

Description of tagset used for POS Tagger:

Stopword

nlp-id also provide list of Indonesian stopword.

from nlp_id.stopword import StopWord 
stopword = StopWord() 
stopword.get_stopword() 
# [{list_of_nlp_id_stopword}]    

Stopword Removal is used to remove every Indonesian stopword from the given text.

from nlp_id.stopword import StopWord 
text = "Lionel Messi pergi Ke pasar di area Jakarta Pusat" # single sentence
stopword = StopWord() 
stopword.remove_stopword(text)
# Lionel Messi pergi pasar area Jakarta Pusat  

paragraph = "Lionel Messi pergi Ke pasar di area Jakarta Pusat itu. Sedangkan Cristiano Ronaldo ke pasar Di area Jakarta Selatan. Dan mereka tidak bertemu begini-begitu."
stopword.remove_stopword(paragraph)
# Lionel Messi pergi pasar area Jakarta Pusat. Cristiano Ronaldo pasar area Jakarta Selatan. bertemu.

Training and Evaluation

Our model is trained using stories from kumparan as the dataset. We managed to get ~93% accuracy on our test set.

Citation