farsflow

farsflow is a lightweight Persian text preprocessing library focused on deterministic, semantic-safe normalization for modern AI and search pipelines.

❓ WHY farsflow??

Persian text often contains inconsistent spacing, Arabic/Persian character variants, invisible Unicode formatting characters, Arabic diacritics, and broken ZWNJ usage that negatively affect search, embeddings, NLP pipelines, and LLM applications.

farsflow provides a minimal and deterministic preprocessing layer designed to clean text without aggressive or semantic-destructive transformations.

---

🚀 Features

• Deterministic and semantic-safe normalization
• Safe ZWNJ (Joiner) correction
• Whitespace and punctuation cleanup
• Line and paragraph structure preserved by default
• Unicode cleanup (Arabic/Persian variants, Bidi controls, invisible formatting characters)
• Configurable normalization options (digits, diacritics, hamza folding, line structure)
• Modular processors
• Zero dependencies

---

📦 Installation

pip install farsflow

✨ Quick Start

import farsflow as ff

text = "سلام  دنیا!  این یك   تست است  که می نویسم  ۴۵۶"
cleaned = ff.clean(text)
print(cleaned)

Expected output:

سلام دنیا! این یک تست است که می‌نویسم 456

---

🧩 Pipeline Components

farsflow ships with a set of modular, composable components:

• Normalizer — character normalization, Unicode cleanup, and optional diacritics removal
• JoinerFixer — fixes ZWNJ usage without over-correction
• SpaceCleaner — trims redundant whitespace and punctuation spacing
• Pipeline — orchestrates components in a deterministic order

You can customize the pipeline:

from farsflow import (
    Pipeline,
    Normalizer,
    SpaceCleaner,
)

pipeline = Pipeline([
    Normalizer(),
    SpaceCleaner(),
    # JoinerFixer skipped to demonstrate modular behavior
])

text = "می  نويسم  که این   يك   متن  تستي است"
cleaned = pipeline(text)
print(cleaned)

Expected output:

می نویسم که این یک متن تستی است

📄 Line and paragraph structure

By default farsflow preserves line breaks. Within each line, runs of spaces and tabs collapse and the edges are trimmed; a run of blank lines becomes a single blank line. This is what you want for RAG chunking, sentence splitting, and anything downstream that relies on paragraph boundaries.

import farsflow as ff

ff.clean("تيتر خبر\n\n\nبراساس گزارش ها، نرخ تورم كاهش يافته است.")
# 'تیتر خبر\n\nبراساس گزارش‌ها، نرخ تورم کاهش یافته است.'

To fold everything onto one line instead — the behavior of 0.1.1 and earlier — set the policy on the Normalizer:

from farsflow import Pipeline, Normalizer, JoinerFixer, SpaceCleaner

pipeline = Pipeline([
    Normalizer(newline_policy="collapse"),
    JoinerFixer(),
    SpaceCleaner(),
])
pipeline("تيتر خبر\n\nمتن خبر")   # 'تیتر خبر متن خبر'

The policy lives on Normalizer — the first stage — because line structure has to be settled before any later rule runs. JoinerFixer deliberately never matches across a line break, so if a later stage turned newlines into spaces, a half-space rule would miss on the first pass and fire on the second, and the output would not be stable.

---

⚠️ Lossy operations are opt-in

Two Normalizer options delete or merge information instead of just reshaping it. Both default to False — ff.clean() never does this unless you ask for it:

• remove_diacritics (default False) — strips Arabic diacritics (harakat, tanvin, shadda, sukun): "کاملاً" → "کاملا". Diacritics are rare in everyday Persian text, but appear in poetry, Quranic quotations, and some literary or TTS text, where deleting them is a real loss. Turn this on for search/embedding pipelines that want maximum recall regardless.
• fold_hamza_carriers (default False) — folds أ/إ to ا and ؤ to و: "مؤلف" → "مولف". These letters carry a real distinction in some Arabic loanwords, so folding them is a lossy, fuzzy-matching choice, not a safe default.

from farsflow import Pipeline, Normalizer, JoinerFixer, SpaceCleaner

pipeline = Pipeline([
    Normalizer(remove_diacritics=True, fold_hamza_carriers=True),
    JoinerFixer(),
    SpaceCleaner(),
])

The ي/ى→ی, ك→ک, and ة/ۀ→ه substitutions stay on unconditionally — those forms are not used distinctly in Persian orthography, so folding them loses nothing.

---

Design Principles

farsflow follows a few core principles:

• deterministic output
• semantic-safe transformations
• opt-in for potentially lossy operations
• zero dependencies
• modular architecture

---

🧪 Testing

pytest
# or:
pytest path/to/test_file.py

---

🗺 Scope

farsflow intentionally stays small: three processors, a pipeline, and a handful of documented options. (v0.1.2)

---

📄 License

MIT License — see LICENSE.

---

🤝 Contributing

Contributions are welcome.
Please open an issue or submit a pull request on GitHub.

📝 Changelog

See CHANGELOG.md for version history.