Skip to main content

Utility library for analysis & (pre)processing of Yorùbá text

Project description

Ìrànlọ́wọ́

Build Status PyPI PyPI - Python Version License Style

Ìrànlọ́wọ́ is a set of utilities to analyze & process Yorùbá text for NLP tasks. The focus is on helping software developers build large, clean text datasets for (further) diacritic restoration and machine translation tasks.

Features

ADR tools

  • Strip all diacritics from word-types
  • Verify that text is NFC or NFD
  • Canonicalize a corpus (from MS Word or elsewhere) → NFC
  • Split long sentences on certain characters like ;,:, etc
  • Automatically restore correct diacritics using a pre-trained model
  • Find all variants of all word-type in a given corpus
  • Partially strip diacritics from word-types

Ready to use webpage scrapers

  • Bíbélì Mímọ́
  • Yoruba Bible - Bible Society of Nigeria
  • Yorùbá Blog
  • BBC Yorùbá

Corpus analysis tools

  • Dataset character distribution
  • Dataset ambuiguity statistics → Lexdif, etc for a given corpus
  • Dataset scoring (proximity to correctly diacritized text, LM perplexity, KL divergence)

Installation

Obtainable from the Python Package Index (PyPI)pip install iranlowo

Example

  • Show computing environment and installation process
  • Diacritize a phrase
$ python
Python 3.7.3 (default, Mar 27 2019, 16:54:48)
[Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import iranlowo.adr as ránlọ
>>> ránlọ.diacritize_text("lootoo ni pe ojo gbogbo ni ti ole")
PRED AVG SCORE: -0.0037, PRED PPL: 1.0037
'lóòtóọ́ ni pé ọjọ́ gbogbo ni ti olè' 
  • Diacritize phrases, note we use ipython only because it renders nicer, easy-to-read text-colours in the terminal!

Disclaimer

This is beta software, if you pass the diacritizer out-of-domain text, English, pidgin or any other non-Yorùbá text, you will experience very marvelous, black-box results.

Since this a work-in-progress and we are steadily improving, if you encounter any problems with correctness or performance, please submit pull-requests with corrections or file an issue.

License

This project is licensed under the MIT License.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

iranlowo-0.0.8.tar.gz (87.9 MB view hashes)

Uploaded Source

Built Distribution

iranlowo-0.0.8-py3-none-any.whl (87.9 MB view hashes)

Uploaded Python 3

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page