Utility library for analysis & (pre)processing of Yorùbá text
Ìrànlọ́wọ́ is a set of utilities to analyze & process Yorùbá text for NLP tasks. The focus is on helping software developers build large, clean text datasets for (further) diacritic restoration and machine translation tasks.
- Strip all diacritics from word-types
- Verify that text is NFC or NFD
- Normalize a corpus (from MS Word or elsewhere) → NFC
- Split long sentences on certain characters like
- Automatically restore correct diacritics using a pre-trained model
- Find all variants of all word-type in a given corpus
- Partially strip diacritics from word-types
Ready to use webpage scrapers
- Bíbélì Mímọ́ (Biblica, Bible Society of Nigeria)
- Yorùbá Blog
- BBC Yorùbá
Corpus analysis tools
- Dataset character distribution
- Dataset ambuiguity statistics → Lexdif, etc for a given corpus
- Dataset scoring (proximity to correctly diacritized text, LM perplexity, KL divergence)
Obtainable from the Python Package Index (PyPI) →
pip install iranlowo
- Show computing environment and installation process
- Diacritize a phrase
$ python Python 3.7.3 (default, Mar 27 2019, 16:54:48) [Clang 4.0.1 (tags/RELEASE_401/final)] :: Anaconda, Inc. on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import iranlowo.adr as ránlọ >>> ránlọ.diacritize_text("lootoo ni pe ojo gbogbo ni ti ole") PRED AVG SCORE: -0.0037, PRED PPL: 1.0037 'lóòtóọ́ ni pé ọjọ́ gbogbo ni ti olè'
- Diacritize phrases, note we use
ipythononly because it renders nicer, easy-to-read text-colours in the terminal!
This is beta software, if you pass the diacritizer out-of-domain text, English, pidgin or any other non-Yorùbá text, you will experience very marvelous, black-box results.
This project is licensed under the MIT License.
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Hashes for iranlowo-0.0.8.3-py3-none-any.whl