Skip to main content

A python package for tokenizing and normalizing texts mainly written in the Igbo Language

Project description

What Is Igbo Text

Igbo Text is a library for tokenizing and normalizing texts mainly written in the Igbo Language.
It is an implementation of the tokenization and normalization algorithm written in the paper Analysis and Representation of Igbo Text Document for a Text-Based System by Ifeanyi-Reuben Nkechi J., Ugwu Chidiebere, Adegbola Tunde.

Installation

$ pip install igbo-text

Examples

Normalization

from igbo_text import IgboText

# Create IgboText class instance
igbo_text = IgboText()

# normalize text 
text = "Ọ nà-ezò nnukwu mmīri n'iro?"
normalized_text = igbo_text.normalize(text, convert_to_lower=True, remove_abbreviations=True)
print(normalized_text)

When the code above is executed, the output will be

ọ na ezo nnukwu mmiri n iro

Upper case characters can be left alone by setting convert_to_lower=False

Abbreviations can be left alone by setting remove_abbreviations=True

Tokenization

from igbo_text import IgboText

# Create IgboText class instance
igbo_text = IgboText()

# tokenize text
text = "Ndị Fàda kwènyèrè n'atọ̀ n'ime otù."
tokenized_text = igbo_text.tokenize(text)
print(tokenized_text)

When the code above isi executed, the output will be

["Ndị", "Fada", "kwenyere", "n'", "atọ", "n'", "ime", "otu", "."]

You can convert all upper case characters to lower case by setting convert_to_lower=True.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for igbo-text, version 0.1.3
Filename, size File type Python version Upload date Hashes
Filename, size igbo_text-0.1.3-py3-none-any.whl (5.5 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size igbo-text-0.1.3.tar.gz (3.9 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page