Skip to main content

The goal of the Indic NLP Library is to build Python based libraries for common text processing and Natural Language Processing in Indian languages.

Project description

Indic NLP Library

The goal of the Indic NLP Library is to build Python based libraries for common text processing and Natural Language Processing in Indian languages. Indian languages share a lot of similarity in terms of script, phonology, language syntax, etc. and this library is an attempt to provide a general solution to very commonly required toolsets for Indian language text.

The library provides the following functionalities:

  • Text Normalization
  • Script Information
  • Word Tokenization and Detokenization
  • Sentence Splitting
  • Word Segmentation
  • Script Conversion
  • Romanization
  • Indicization
  • Transliteration
  • Translation

The data resources required by the Indic NLP Library are hosted in a different repository. These resources are required for some modules. You can download from the Indic NLP Resources project.

If you are interested in Indian language NLP resources, you should check the Indic NLP Catalog for pointers.

Pre-requisites

Configuration

  • Add the project to the Python Path:

    export PYTHONPATH=$PYTHONPATH:<project base directory>/src

  • Export the path to the Indic NLP Resources directory

    export INDIC_RESOURCES_PATH=<path to Indic NLP resources>

Usage

Website

http://anoopkunchukuttan.github.io/indic_nlp_library

Author

Anoop Kunchukuttan ( anoop.kunchukuttan@gmail.com )

Version: 0.6

Revision Log

0.6 : 16 Dec 2019:

- New romanizer and indicizer
- Script Unifiers
- Improved script normalizers
- Added contrib directory for sample uses
- changed to MIT license 

0.5 : 03 Jun 2019:

- Improved word tokenizer to handle dates and numbers. 
- Added sentence splitter that can handle common prefixes/honorofics and uses some heuristics.
- Added detokenizer
- Added acronym transliterator that can convert English acronyms to Brahmi-derived scripts

0.4 : 28 Jan 2019: Ported to Python 3, and lots of feature additions since last release; primarily around script information, script similarity and syllabification.

0.3 : 21 Oct 2014: Supports morph-analysis between Indian languages

0.2 : 13 Jun 2014: Supports transliteration between Indian languages and tokenization of Indian languages

0.1 : 12 Mar 2014: Initial version. Supports text normalization.

LICENSE

Indic NLP Library is released under the MIT license

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

indic_nlp_library-0.6.tar.gz (25.2 kB view details)

Uploaded Source

Built Distribution

indic_nlp_library-0.6-py3-none-any.whl (34.4 kB view details)

Uploaded Python 3

File details

Details for the file indic_nlp_library-0.6.tar.gz.

File metadata

  • Download URL: indic_nlp_library-0.6.tar.gz
  • Upload date:
  • Size: 25.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.4.2 requests/2.22.0 setuptools/42.0.2 requests-toolbelt/0.9.1 tqdm/4.40.2 CPython/3.6.5

File hashes

Hashes for indic_nlp_library-0.6.tar.gz
Algorithm Hash digest
SHA256 075b12ec88a56a434edd7202801e9411ddcb5547b144ac3d9c52ca8031f55b14
MD5 996ca2613546cf0d49bdcaf1b619d4ea
BLAKE2b-256 98751c352e82e1ce8467a5e0af0b0594c6dd81dd131818f2529f34e4cb4b4228

See more details on using hashes here.

Provenance

File details

Details for the file indic_nlp_library-0.6-py3-none-any.whl.

File metadata

  • Download URL: indic_nlp_library-0.6-py3-none-any.whl
  • Upload date:
  • Size: 34.4 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.4.2 requests/2.22.0 setuptools/42.0.2 requests-toolbelt/0.9.1 tqdm/4.40.2 CPython/3.6.5

File hashes

Hashes for indic_nlp_library-0.6-py3-none-any.whl
Algorithm Hash digest
SHA256 862ddc04b9ad45c68bb792ce1dd410f97cf9a01937b93b0b9445f5d2253edd42
MD5 fd00cc2dc935fb8261bbc9768f801d03
BLAKE2b-256 f0588d1e621f87bbc4217fb8ce6628a2eb08b65a64582c5531becf41da5d721c

See more details on using hashes here.

Provenance

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page