farsflow 0.1.0

Fast, modern, and modular Persian text preprocessing library.

farsflow

farsflow is a small library for Persian text preprocessing, designed for practical NLP and LLM workflows.

---

🚀 Features

• Deterministic pipeline (same input -> same output)
• Safe character normalization
• Joiner (ZWNJ) fixes
• Whitespace & punctuation cleanup
• Modular processors (use only the components you need)
• Real‑world style sample tests
• Zero dependencies

---

📦 Installation

pip install farsflow

✨ Quick Start

import farsflow as ff

text = "سلام  دنیا!  این یك   تست است  که می نویسم  ۴۵۶"
cleaned = ff.clean(text)
print(cleaned)

Expected output: سلام دنیا! این یک تست است که می‌نویسم 456

---

🧩 Pipeline Components

farsflow ships with a set of modular, composable components:

• Normalizer — safe character normalization
• JoinerFixer — fixes ZWNJ usage without over-correction
• SpaceCleaner — trims redundant whitespace and punctuation spacing
• Pipeline — orchestrates components in a deterministic order

You can customize the pipeline:

from farsflow import Pipeline, Normalizer, JoinerFixer

pipeline = Pipeline(
    Normalizer(),
    JoinerFixer(),
)

text = "متن   تستی"
pipeline(text)

---

🧪 Testing

pytest
# or:
pytest path/to/test_file.py

---

🗺 Roadmap (v0.2.0)

• formalize the behavior of "ff.clean" as a safe and deterministic baseline
• add optional normalization controls (e.g. digit normalization)
• add optional noise-cleaning emoji and url processors
• introduce simple profiles (ff.llm.clean, ff.embedding.query, ff.embedding.index) built on top of the same core
• expand real-world test cases to ensure stable behavior across informal, mixed, and noisy Persian text

---

📄 License

MIT License — see LICENSE.

---

🤝 Contributing

Contributions are welcome.
Please open an issue or submit a pull request on GitHub.

📝 Changelog

See CHANGELOG for version history.