A python package for tokenizing and normalizing texts mainly written in the Igbo Language
Project description
What Is Igbo Text
Igbo Text is a library for tokenizing and normalizing texts mainly written in the Igbo Language.
It is an implementation of the tokenization and normalization algorithm written in the paper Analysis and Representation of Igbo Text Document for a Text-Based System by Ifeanyi-Reuben Nkechi J., Ugwu Chidiebere, Adegbola Tunde.
Installation
$ pip install igbo-text
Examples
Normalization
from igbo-text import IgboText
# Create IgboText class instance
igbo_text = IgboText()
# normalize text
text = "Ọ nà-ezò nnukwu mmīri n'iro?"
normalized_text = igbo_text.normalize(text, convert_to_lower=True, remove_abbreviations=True)
print(normalized_text)
When the code above is executed, the output will be
ọ na ezo nnukwu mmiri n iro
Upper case characters can be left alone by setting convert_to_lower=False
Abbreviations can be left alone by setting remove_abbreviations=True
Tokenization
from igbo-text import IgboText
# Create IgboText class instance
igbo_text = IgboText()
# tokenize text
text = "Ndị Fàda kwènyèrè n'atọ̀ n'ime otù."
tokenized_text = igbo_text.tokenize(text)
print(tokenized_text)
When the code above isi executed, the output will be
["Ndị", "Fada", "kwenyere", "n'", "atọ", "n'", "ime", "otu", "."]
You can convert all upper case characters to lower case by setting convert_to_lower=True.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for igbo_text-0.1.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 01fee325cd38c74a81a0e2b9d710286d8929b4e855a6e7510d198369e532dc17 |
|
MD5 | 637057229d50e3656b2cca3800137e96 |
|
BLAKE2b-256 | fa981c375838a704e12bcf9abe4979bc547f074753d8dbdce36111b06f525dfc |