farsflow 0.1.1

Deterministic, semantic-safe Persian text preprocessing for AI, NLP, and search pipelines.

farsflow

farsflow is a lightweight Persian text preprocessing library focused on deterministic, semantic-safe normalization for modern AI and search pipelines.

❓ WHY farsflow??

Persian text often contains inconsistent spacing, Arabic/Persian character variants, invisible Unicode formatting characters, Arabic diacritics, and broken ZWNJ usage that negatively affect search, embeddings, NLP pipelines, and LLM applications.

farsflow provides a minimal and deterministic preprocessing layer designed to clean text without aggressive or semantic-destructive transformations.

---

🚀 Features

• Deterministic and semantic-safe normalization
• Safe ZWNJ (Joiner) correction
• Whitespace and punctuation cleanup
• Unicode cleanup (Arabic/Persian variants, Bidi controls, invisible formatting characters)
• Configurable normalization options (digit normalization & diacritics removal)
• Modular processors
• Zero dependencies

---

📦 Installation

pip install farsflow

✨ Quick Start

import farsflow as ff

text = "سلام  دنیا!  این یك   تست است  که می نویسم  ۴۵۶"
cleaned = ff.clean(text)
print(cleaned)

Expected output:

سلام دنیا! این یک تست است که می‌نویسم 456

---

🧩 Pipeline Components

farsflow ships with a set of modular, composable components:

• Normalizer — character normalization, Unicode cleanup, and optional diacritics removal
• JoinerFixer — fixes ZWNJ usage without over-correction
• SpaceCleaner — trims redundant whitespace and punctuation spacing
• Pipeline — orchestrates components in a deterministic order

You can customize the pipeline:

from farsflow import (
    Pipeline,
    Normalizer,
    SpaceCleaner,
)

pipeline = Pipeline([
    Normalizer(),
    SpaceCleaner(),
    # JoinerFixer skipped to demonstrate modular behavior
])

text = "می  نويسم  که این   يك   متن  تستي است"
cleaned = pipeline(text)
print(cleaned)

Expected output:

می نویسم که این یک متن تستی است

Design Principles

farsflow follows a few core principles:

• deterministic output
• semantic-safe transformations
• opt-in for potentially lossy operations
• zero dependencies
• modular architecture

---

🧪 Testing

pytest
# or:
pytest path/to/test_file.py

---

🗺 Roadmap (v0.2.0)

• configurable preprocessing profiles for different use cases
• optional emoji and URL cleanup processors
• compound word normalization (opt-in)
• additional normalization rules based on real-world corpora
• performance benchmarking and optimization

---

📄 License

MIT License — see LICENSE.

---

🤝 Contributing

Contributions are welcome.
Please open an issue or submit a pull request on GitHub.

📝 Changelog

See CHANGELOG for version history.