IBITNormalizer
Simple persian text-normalizer base on hazm lib
install
pip install IBITNormalizer --upgrade
import
from IBITNormalizer.normalizer import IBITNormalizer
for lm task
text = """
سلام خوبی
از بیرون چخبر
چیکارا میکنی
تازگیا هوا چقدر سرد شده نه ؟
"""
normalizer = IBITNormalizer.forLM()
print("forLM -> ", normalizer.normalize(text))
output:
forLM -> سلام خوبی
از چخبر
چیکارا میکنی
تازگیا هوا سرد نه؟
for llm task
text = """
سلام خوبی
از بیرون چخبر
چیکارا میکنی
تازگیا هوا چقدر سرد شده نه ؟
"""
normalizer = IBITNormalizer.forLLM()
print("forLLM -> ", normalizer.normalize(text))
output:
forLLM -> سلام خوبی
از بیرون چخبر
چیکارا میکنی
تازگیا هوا چقدر سردشدهنه؟
1 maintainer
Release files for IBITNormalizer 1.2.8
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| IBITNormalizer-1.2.8.tar.gz | 23.8 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| IBITNormalizer-1.2.8-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 49.9 kB
Release files / IBITNormalizer-1.2.8.tar.gz
| Download URL | IBITNormalizer-1.2.8.tar.gz |
|---|---|
| Size | 23.8 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
ce880d9d0b6b98a53431ee4ff84d1f09c76cc44e219da4c43bd4ccc156b35ccf
|
|
BLAKE2b-256 checksum How to use checksums |
045772aa2dc314229c0f8c8b03e08788afda174f686671fa485bec86c1ba6581
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.0.0 CPython/3.11.2
|
Release files / IBITNormalizer-1.2.8-py3-none-any.whl
| Download URL | IBITNormalizer-1.2.8-py3-none-any.whl |
|---|---|
| Size | 26.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
6e30fcd782ce0d155aa0a2d35b74e3d1c519bc72f519bd6b631fef539a3657f6
|
|
BLAKE2b-256 checksum How to use checksums |
5441958b15b0ecb886d3e6640dbc335ed11dd78f5059379bd5bb049b31294c51
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.0.0 CPython/3.11.2
|
