araclean

Arabic text normalization and cleaning — pure-Python, composable, reproducible, offset-preserving.

Status: pre-release (0.x). The normalization core is tested, but the API may still change before 1.0.

araclean is non-destructive by default: the bare call is lossless encoding repair (Unicode form, presentation forms, tatweel, bidi/zero-width characters, look-alike letters, whitespace) and never silently strips tashkeel or folds letters. Everything lossy is opt-in through named, serializable profiles (SEARCH, ML, SOCIAL, CLASSICAL), so the exact preprocessing a corpus went through can be published and reproduced.

>>> from araclean import normalize
>>> normalize("على قدر أهل العـــزم")                 # default: remove tatweel
'على قدر أهل العزم'
>>> normalize("عَلَى قَدْرِ أَهْلِ العَزْمِ", profile="search")
'علي قدر اهل العزم'

For span-level work — RAG citation, NER projection — apply_aligned returns the normalized text and a map back to every original position:

>>> from araclean import Pipeline, RemoveTatweel, FoldAlef
>>> pipe = Pipeline([RemoveTatweel(), FoldAlef()])
>>> normalized, omap = pipe.apply_aligned("أحمـد")
>>> normalized
'احمد'
>>> omap.to_original((0, 4))   # where does the whole normalized word sit in the original?
(0, 5)

See the map in action in the live Hugging Face Space, or read the offset-preserving normalization guide.

Documentation

Full documentation lives at https://mhdmartini.github.io/araclean/:

• Getting started — install, first call, choosing a profile.
• Profiles — every step each profile applies, lossless vs lossy (generated from the code).
• Guides — the CLI, pandas & polars, tuning and composing pipelines, custom steps, reproducibility, and stopwords.
• Why araclean — the problem it solves and its design choices.
• API reference and CLI reference.

CI runs every Python example as a doctest and checks generated pages (profiles, glossary, and CLI reference) against the code.

Install

pip install araclean

Optional extras:

pip install "araclean[cli]"     # command-line interface
pip install "araclean[pandas]"  # pandas Series accessor
pip install "araclean[polars]"  # polars accessor
pip install "araclean[all]"     # everything

The core install is lean: it requires only pydantic v2 — no compiler, Java, or data download.

Development

This project uses uv for environments and pre-commit for the quality gate.

uv sync                       # create the dev environment
uv run pre-commit install     # wire up the pre-commit hooks
uv run pre-commit run --all-files

The gate runs ruff, mypy --strict, pyright, pytest, and cspell (canonical Arabic terminology per GLOSSARY.md).

Commits & versioning

The version and changelog are derived from commit messages by Commitizen — see ADR-0008. Every commit must be a Conventional Commit; the format is enforced by a commit-msg hook (wired by uv run pre-commit install) and in CI on every PR.

feat(steps): add RemoveTashkeel step     # → minor bump
fix(pipeline): preserve step order        # → patch bump
feat(api)!: rename normalize() argument   # breaking → minor (we are pre-1.0)

feat bumps the minor, fix/others the patch, and a breaking change (! or a BREAKING CHANGE: footer) the minor — the project stays in 0.x until 1.0 is declared. To cut a release:

uv run cz bump        # compute the bump, update pyproject.toml + uv.lock + CHANGELOG.md, tag vX.Y.Z
uv run cz changelog   # preview release notes without bumping

Never hand-edit [project].version — Commitizen owns it.

License

MIT