The goal of the Indic NLP Library is to build Python based libraries for common text processing and Natural Language Processing in Indian languages.
Project description
Indic NLP Library
The goal of the Indic NLP Library is to build Python based libraries for common text processing and Natural Language Processing in Indian languages. Indian languages share a lot of similarity in terms of script, phonology, language syntax, etc. and this library is an attempt to provide a general solution to very commonly required toolsets for Indian language text.
The library provides the following functionalities:
- Text Normalization
- Script Information
- Word Tokenization and Detokenization
- Sentence Splitting
- Word Segmentation
- Script Conversion
- Romanization
- Indicization
- Transliteration
- Translation
The data resources required by the Indic NLP Library are hosted in a different repository. These resources are required for some modules. You can download from the Indic NLP Resources project.
If you are interested in Indian language NLP resources, you should check the Indic NLP Catalog for pointers.
Pre-requisites
- Python 3.x (For Python 2.x version check the tag
PYTHON_2.7_FINAL_JAN_2019
. Not actively supporting Python 2.x anymore, but will try to maintain as much compatibility as possible) - Morfessor 2.0 Python Library
- Indic NLP Resources
Configuration
-
Add the project to the Python Path:
export PYTHONPATH=$PYTHONPATH:<project base directory>/src
-
Export the path to the Indic NLP Resources directory
export INDIC_RESOURCES_PATH=<path to Indic NLP resources>
Usage
- Python API: Check this IPython Notebook for examples.
- You can find the Python 2.x Notebook here
- Commandline Interface: The commandline interface is documented on the project website
Website
http://anoopkunchukuttan.github.io/indic_nlp_library
Author
Anoop Kunchukuttan ( anoop.kunchukuttan@gmail.com )
Version: 0.6
Revision Log
0.6 : 16 Dec 2019:
- New romanizer and indicizer
- Script Unifiers
- Improved script normalizers
- Added contrib directory for sample uses
- changed to MIT license
0.5 : 03 Jun 2019:
- Improved word tokenizer to handle dates and numbers.
- Added sentence splitter that can handle common prefixes/honorofics and uses some heuristics.
- Added detokenizer
- Added acronym transliterator that can convert English acronyms to Brahmi-derived scripts
0.4 : 28 Jan 2019: Ported to Python 3, and lots of feature additions since last release; primarily around script information, script similarity and syllabification.
0.3 : 21 Oct 2014: Supports morph-analysis between Indian languages
0.2 : 13 Jun 2014: Supports transliteration between Indian languages and tokenization of Indian languages
0.1 : 12 Mar 2014: Initial version. Supports text normalization.
LICENSE
Indic NLP Library is released under the MIT license
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
File details
Details for the file indic_nlp_library-0.6.tar.gz
.
File metadata
- Download URL: indic_nlp_library-0.6.tar.gz
- Upload date:
- Size: 25.2 kB
- Tags: Source
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.4.2 requests/2.22.0 setuptools/42.0.2 requests-toolbelt/0.9.1 tqdm/4.40.2 CPython/3.6.5
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 075b12ec88a56a434edd7202801e9411ddcb5547b144ac3d9c52ca8031f55b14 |
|
MD5 | 996ca2613546cf0d49bdcaf1b619d4ea |
|
BLAKE2b-256 | 98751c352e82e1ce8467a5e0af0b0594c6dd81dd131818f2529f34e4cb4b4228 |
Provenance
File details
Details for the file indic_nlp_library-0.6-py3-none-any.whl
.
File metadata
- Download URL: indic_nlp_library-0.6-py3-none-any.whl
- Upload date:
- Size: 34.4 kB
- Tags: Python 3
- Uploaded using Trusted Publishing? No
- Uploaded via: twine/3.1.1 pkginfo/1.4.2 requests/2.22.0 setuptools/42.0.2 requests-toolbelt/0.9.1 tqdm/4.40.2 CPython/3.6.5
File hashes
Algorithm | Hash digest | |
---|---|---|
SHA256 | 862ddc04b9ad45c68bb792ce1dd410f97cf9a01937b93b0b9445f5d2253edd42 |
|
MD5 | fd00cc2dc935fb8261bbc9768f801d03 |
|
BLAKE2b-256 | f0588d1e621f87bbc4217fb8ce6628a2eb08b65a64582c5531becf41da5d721c |