LIAS-Labrary 0.2.0

A lightweight Python library for preprocessing Arabic dialect text written in Arabic script.

LIAS_Labrary

LIAS_Labrary is a lightweight Python library for preprocessing Arabic dialect text written in Arabic script.
It is designed to support both:

• single-text preprocessing
• dataset / DataFrame column preprocessing

The library focuses on safe and practical preprocessing operations commonly needed in Arabic dialect NLP and sentiment analysis workflows, while preserving the original Arabic letter forms.

---

Main Features

LIAS_Labrary currently supports:

Core text preprocessing

• Remove Arabic diacritics
• Remove tatweel (ـ)
• Reduce repeated letters
• Reduce repeated punctuation
• Normalize whitespace
• Remove or preserve selected special characters
• Convert digits between:
  • Western digits: 0 1 2 3 4 5 6 7 8 9
  • Eastern Arabic digits: ٠ ١ ٢ ٣ ٤ ٥ ٦ ٧ ٨ ٩

Emoji handling

• keep
• remove
• replace

When using replace, users can choose:

• replacement position:
  • inline
  • append
  • prepend
• output format:
  • plain
  • tagged

URL handling

• keep
• remove
• replace

When using replace, users can choose:

• replacement position:
  • inline
  • append
  • prepend
• output format:
  • plain
  • tagged
• replacement token:
  • default: رابط
  • customizable to any language or token

Hashtag handling

• keep
• normalize
• remove

Example:

• #هاد الفيلم زوين → هاد الفيلم زوين

HTML cleaning

• Remove HTML tags for web scraping / crawling / extracted text scenarios

Dataset-level preprocessing

For each operation, LIAS_Labrary also provides *_data functions that:

• process a selected column in a pandas DataFrame
• preserve the full dataset
• return the dataset with the processed column added or updated

---

Design Principles

LIAS_Labrary follows a conservative preprocessing philosophy:

• It preserves original Arabic letters
• It does not perform aggressive Arabic letter normalization
• It avoids transformations such as:
  • أ / إ / آ → ا
  • ى → ي
  • ة → ه
  • simplification of ؤ / ئ

This choice helps preserve orthographic information for:

• pretrained language models
• later lemmatization
• future transcoding or advanced linguistic processing

---

Installation

From PyPI

pip install LIAS_Labrary