Xina processing library
Project description
Xina Processor is an open source library for cleaning Arabic text. It includes various cleaning functions as well as modules for streaming file and folder cleaning.
Installation
PIP
If you use pip, you can install xinaprocessor with:
pip install xinaprocessor
From source
You can directly clone this repo and install the library. First clone the repo with:
git clone https://github.com/xina-ai/xinaprocessor.git
Then cd to the directory and install the library with:
pip install -e .
Documentation
Documentation is still in process here: https://xina-ai.github.io/xinaprocessor/
Getting Started
from xinaprocessor import cleaners
To clean text
Text = "نص عربي!"
Cleaner = cleaners.TextCleaner(text=Text)
Cleaner.keep_arabic_only()
To clean text File
# Creating File MyData.txt
FilePath = "MyData.txt"
with open(FilePath, "w") as f:
f.write("Aالسطر الأول\nالسطر الثاني!")
# Creating FileCleaner object
Cleaner = cleaners.FileCleaner(filepath=FilePath)
Cleaner.remove_english_text().remove_arabic_numbers().remove_punctuations()
# To access the resulted data
CleanedData = Cleaner.lines # the result will look like ['السطر الأول', 'السطر الثاني']
CleanedText = Cleaner.text # the result will look like 'السطر الأول\nالسطر الثاني'
# To save the proccessed/cleaned text to a file
Cleaner.save2file('CleanedData.txt', encoding='utf-8')
To clean large text File
# This Cleaner is used for large text files, the cleaned texts will be saved to CleanedFile.txt file
FilePath = "MyData.txt"
CleanedPath = "CleanedFile.txt"
Cleaner = cleaners.FileStreamCleaner(filepath=FilePath, savepath=CleanedPath)
Cleaner.remove_hashtags().remove_honorific_signs().drop_empty_lines().clean()
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
xinaprocessor-0.4.tar.gz
(16.9 kB
view hashes)
Built Distribution
Close
Hashes for xinaprocessor-0.4-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | d748215f3ed56c08ed1be45a213d69e9915c194e9ced582fabdf914de3bed19a |
|
MD5 | 00b1d969d5ef37be6852fdc9217f128c |
|
BLAKE2b-256 | 88ecc805fa7e4476ab2d6577b9b7a6532a0ded5c680bb5c65f5701eca255ec04 |