urduflow

Urdu-aware primitives for LLM/RAG pipelines.

Most LLM infrastructure assumes English punctuation, Latin script, and monolingual queries. Pakistani users often mix Urdu script, Roman Urdu, English technical terms, and numbers in the same message.

urduflow helps with:

1. Script normalization — Arabic/Urdu Unicode variants collapse to stable canonical forms before embedding.
2. Morphology-aware tokenization — conservative splitting of common Urdu suffixes and postpositions.
3. Urdu smart chunking — respects Urdu punctuation (۔ ؟ ؛), poetry/verse lines, and closing punctuation.
4. Code-switch routing — labels Urdu, Roman Urdu, English, numbers, and mixed spans.
5. Framework adapters — optional LangChain and LlamaIndex-compatible splitters.

Zero runtime dependencies. Python 3.9+.

pip install urduflow

Or install from GitHub:

pip install git+https://github.com/GurusGeek/urduflow.git

Why this exists

Common RAG pipeline failures with Urdu text:

• كتاب and کتاب can embed differently even though they represent the same word in different Unicode conventions.
• Western splitters often ignore Urdu sentence endings (۔, ؟, ؛).
• Poetry and verse can get flattened into prose chunks.
• Queries like mera code run nahi ho raha can you fix the Python error are code-switched, not purely English or Urdu script.

urduflow is a small, auditable toolkit for those gaps. It is not a full Urdu NLP stack. For lemmatization, POS tagging, or broader NLP tasks, see projects like urduhack, LughaatNLP, and urdu-nlp. This package focuses on Urdu-aware middleware for LLM/RAG pipelines: normalization, chunking, code-switch routing, optional framework adapters, and seed benchmarks.

Quick start

from urduflow import UrduNormalizer, UrduSmartSplitter, CodeSwitchRouter

normalizer = UrduNormalizer()
print(normalizer.normalize("كتاب"))
# کتاب

splitter = UrduSmartSplitter(chunk_size=80, chunk_overlap=10)
chunks = splitter.split_text("یہ پہلا جملہ ہے۔ یہ دوسرا جملہ ہے۔")
print(chunks)

router = CodeSwitchRouter()
print(router.route("mera code run nahi ho raha can you fix the Python error"))
# [{'span': 'roman_urdu', 'text': 'mera'},
#  {'span': 'english', 'text': 'code run'},
#  {'span': 'roman_urdu', 'text': 'nahi ho raha'},
#  {'span': 'english', 'text': 'can you fix the Python error'}]

Modules

UrduNormalizer

from urduflow import normalize, canonical, similarity_key

normalize("كتاب")       # کتاب
canonical("كَتَابٍ")    # کتاب
similarity_key("  كتاب ") # کتاب

Normalizes:

• Arabic kaf ك → Urdu kaf ک
• Arabic yeh ي / alif maqsura ى → Urdu yeh ی
• Arabic digits ١٢٣ → Urdu digits ۱۲۳
• zero-width characters removed
• canonical form strips harakat/diacritics

UrduMorphTokenizer

from urduflow import UrduMorphTokenizer

tok = UrduMorphTokenizer()
print(tok.tokenize("کتابوں"))
# ['کتاب', 'وں']

Conservative suffix handling. It is not a full morphological analyzer; it provides safer building blocks for LLM/RAG preprocessing.

UrduSmartSplitter

from urduflow import UrduSmartSplitter

s = UrduSmartSplitter(chunk_size=60, chunk_overlap=10, respect_verse=True)
print(s.split_text("پہلی سطر\nدوسری سطر\nتیسری سطر"))
# ['پہلی سطر', 'دوسری سطر', 'تیسری سطر']

Features:

• Urdu sentence endings: ۔, ؟, ؛
• ASCII endings: ., !, ?
• decimal safety (3.14 not split)
• whole-word overlap
• no orphaned closing punctuation
• verse/newline preservation
• metadata helper: split_with_metadata()

CodeSwitchRouter

from urduflow import CodeSwitchRouter

r = CodeSwitchRouter()
r.dominant("mera code nahi chal raha")
# roman_urdu

r.needs_bilingual_model("mera code run nahi ho raha can you fix")
# True

Uses a lexicon for Roman Urdu detection. Unknown Latin tokens are treated as English.

LangChain-compatible adapter

from urduflow.integrations import UrduFlowTextSplitter

splitter = UrduFlowTextSplitter(chunk_size=500, chunk_overlap=50)
docs = splitter.create_documents(["یہ پہلا جملہ ہے۔ یہ دوسرا جملہ ہے۔"])

LangChain is optional. If it is installed, returned documents use LangChain's Document; otherwise create_documents() returns plain dictionaries with page_content and metadata.

RomanUrduTransliterator

from urduflow import RomanUrduTransliterator

RomanUrduTransliterator().transliterate("mera kitab")
# میرا کتاب

Lexicon-based transliteration. Unknown words pass through unchanged.

Benchmarks

Seed benchmarks live in benchmarks/. They are small, reproducible checks — not large-scale or embedding-model evaluations.

python benchmarks/eval_codeswitch.py benchmarks/code_switch_200.jsonl
python benchmarks/eval_chunking.py
python benchmarks/eval_pdf_rag.py
python benchmarks/eval_normalization_retrieval.py

Local results on the current seed datasets:

• Code-switch route sequence: 302/326 (92.64%) on code_switch_200.jsonl
• Urdu PDF-RAG seed retrieval: urduflow 20/20 vs naive ASCII splitter 17/20
• Normalization retrieval seed: normalized 8/8 vs raw 7/8
• Urdu boundary detection: urduflow passes both seed cases; naive ASCII punctuation splitter fails both

More research notes and raw logs are in research/.

CLI

urduflow normalize "كتاب"
urduflow canonical "كَتَابٍ"
urduflow split --chunk-size 80 "یہ پہلا جملہ ہے۔ یہ دوسرا جملہ ہے۔"
urduflow route "mera code nahi chal raha can you fix"
urduflow transliterate "mera masla yeh hai"
urduflow benchmark codeswitch --dataset benchmarks/code_switch_200.jsonl

Examples and notebooks

See examples/ for executable demos and notebooks/ for minimal LangChain/LlamaIndex walkthroughs.

Development

pip install -e ".[dev]"
python -m pytest tests/ -q

License

MIT