Skip to main content

No project description provided

Project description

Piraye: NLP Utils

PyPI Version Python Versions License Pylint Unit Test

A utility for normalizing persian, arabic and english texts

Requirements

  • Python 3.9+
  • nltk 3.4.5+

Installation

Install the latest version with pip pip install piraye

Usage

Create an instance of Normalizer with NormalizerBuilder and then call normalize function. Also see list of all available configs in configs section.

  • Using builder pattern:
from piraye import NormalizerBuilder
from piraye.normalizer_builder import Config

text = "این یک متن تسة اسﺘ       , 24/12/1400 "
normalizer = NormalizerBuilder().alphabet_fa().digit_fa().punctuation_fa().tokenizing().remove_extra_spaces().build()
normalizer.normalize(text)  # "این یک متن تست است ، ۲۴/۱۲/۱۴۰۰"
  • Using constructor:
from piraye import NormalizerBuilder
from piraye.normalizer_builder import Config

text = "این یک متن تسة اسﺘ       , 24/12/1400 "
normalizer = NormalizerBuilder([Config.PUNCTUATION_FA, Config.ALPHABET_FA, Config.DIGIT_FA], remove_extra_spaces=True, tokenization=True).build()
normalizer.normalize(text)  # "این یک متن تست است ، ۲۴/۱۲/۱۴۰۰"

Also see other examples

Configs

Config Function Description
ALPHABET_AR alphabet_ar mapping alphabet characters to arabic
ALPHABET_EN alphabet_en mapping alphabet characters to english
ALPHABET_FA alphabet_fa mapping alphabet characters to persian
DIGIT_AR digit_ar convert digits to arabic digits
DIGIT_EN digit_en convert digits to english digits
DIGIT_FA digit_fa convert digits to persian digits
DIACRITIC_DELETE diacritic_delete remove all diacritics
SPACE_DELETE space_delete remove all spaces
SPACE_NORMAL space_normal normal spaces ( like NO-BREAK SPACE , Tab and etc...)
SPACE_KEEP space_keep mapping spaces and not normal them
PUNCTUATION_AR punctuation_ar mapping punctuations to arabic punctuations
PUNCTUATION_Fa punctuation_fa mapping punctuations to persian punctuations
PUNCTUATION_EN punctuation_en mapping punctuations to english punctuations

Other attributes:

  • remove_extra_spaces : append multiple spaces together
  • tokenization : replace punctuation characters that just are tokens

License

GNU Lesser General Public License v2.1

Primarily used for software libraries, the GNU LGPL requires that derived works be licensed under the same license, but works that only link to it do not fall under this restriction. There are two commonly used versions of the GNU LGPL.

See LICENSE

About ️

Arusha

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

piraye-0.3.0.tar.gz (45.6 kB view details)

Uploaded Source

Built Distribution

piraye-0.3.0-py3-none-any.whl (48.8 kB view details)

Uploaded Python 3

File details

Details for the file piraye-0.3.0.tar.gz.

File metadata

  • Download URL: piraye-0.3.0.tar.gz
  • Upload date:
  • Size: 45.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.9.16

File hashes

Hashes for piraye-0.3.0.tar.gz
Algorithm Hash digest
SHA256 ac56e53f4b4c48c68325129856527c9d98354b4fb1cd51f16d500c1cbdb4c7a7
MD5 269b6492cccb551d99797ed1af9f647d
BLAKE2b-256 eac3845f9c71f0fb2c0939436b299101148b55598c3030e11637afa18ca94401

See more details on using hashes here.

File details

Details for the file piraye-0.3.0-py3-none-any.whl.

File metadata

  • Download URL: piraye-0.3.0-py3-none-any.whl
  • Upload date:
  • Size: 48.8 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/4.0.2 CPython/3.9.16

File hashes

Hashes for piraye-0.3.0-py3-none-any.whl
Algorithm Hash digest
SHA256 57cb1ac796d301afeb3b270cc20cd2bcd204825270ac5fab36daef96ecedfa56
MD5 ed2a586e3293b5d5c3ad96de9ca0ef3c
BLAKE2b-256 f06c2c6f29897decf0db2868a97f15954385c07e0db63fae17f00f9961d2f8c3

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page