Rishe (ریشه)

Fast Persian NLP and morphology, built in Rust.

Rishe (root in Persian) turns raw Persian text into clean, structured data in one call: normalized text, tokens, lemmas, sentences, ezafe markers, and a formal rewrite of colloquial writing. It also ships the morphology utilities that usually live in a separate library: stemming, verb conjugation, inflection, token splitting, spell correction, and stopword removal.

The core is Rust. You get it as a pip install with prebuilt wheels, no runtime dependencies, and the same engine as a Rust crate. The base pipeline needs no model downloads; optional POS/NER/ezafe packs are separate.

from rishe import Pipeline

pipe = Pipeline()
doc = pipe("اون داره تو خونه کتاب میخونه، چیه؟")

doc.tokens       # ['آن', 'دارد', 'تو', 'خانه', 'کتاب', 'می‌خواند', '،', 'چیست', '؟']
doc.lemmas       # ['آن', 'داشت', 'تو', 'خانه', 'کتاب', 'خواند', '،', 'چیست', '؟']
doc.formal_text  # 'آن دارد تو خانه کتاب می‌خواند، چیست؟'

Why Rishe

• Normalization about 80× faster than Hazm and Shekar: 1.2 µs per sentence against 96 µs and 104 µs.
• Best tokenization and lemmatization in this comparison. Token F1 of 1.0000 on UD-Seraji, 0.9989 on UD-PerDT and 0.9028 on PEYMA; lemma accuracy of 0.9209 and 0.9316, ahead of Hazm (0.9048 / 0.9009) and Shekar (0.81 / 0.82).
• Morphology in the same core. Stemming, conjugation, inflection, token splitting, spell correction and stopwords — no second library, no extra runtime. On the UD lemma proxy, stemming is 89.2% / 89.5% exact against Hazm (66.4% / 64.4%) and Shekar (76.1% / 72.0%).
• Spell correction that is fast enough to stay in the loop. 80.8% top-1 and 96.2% top-5 on synthetic 1-edit typos, at 0.16 ms per word; Shekar's statistical checker is 60.8% / 77.6% at 94 ms per word on the same set.
• Embeddings without a Python runtime. A 2.3 GB fastText .vec loads in 5.9 s (Hazm/gensim: 176 s) with bit-identical vectors; sentence pooling is 2.5× faster, nearest neighbours match Hazm's top-10, and batch calls release the GIL.
• POS, NER and ezafe without a Python ML stack. Three compact task packs load through ModelPack into the pure-Rust ONNX runtime; no ONNX Runtime, transformers or torch install, and inference releases the GIL.
• Millions of tokens per second. 1.0M tokens/s on one thread; pipe() runs batches in parallel with the GIL released and reaches 6.5M tokens/s on 12 threads.
• Everything in one call. Normalization, informal → formal conversion, tokenization, lemmatization, sentence splitting and ezafe detection from one Pipeline object, in about 16 µs per sentence.
• Informal Persian handled. Colloquial forms (اون, خونه, داره, میخوام, چیه) are rewritten to their formal spelling, and fused clitics such as بهش and ازش are split into به + او and از + او.
• Nothing to set up. Compiled-in lexicons (190k nouns, 1.2k verb stems, word frequencies, stopwords), no Python dependencies, and no network access for the base pipeline. Optional task packs are separate downloads.
• Wheels for every major platform. Linux (glibc and musl, x86_64 and aarch64), macOS (Intel and Apple Silicon) and Windows, for Python 3.9+ (abi3). The same core is available as a Rust crate.
• Thread-safe. One Pipeline can be shared across threads.

Benchmarks

Measured against Hazm and Shekar on the full UD Persian-Seraji, UD Persian-PerDT and PEYMA test sets. Every library is built once and reused.

Core speed (per sentence, single thread)

Core accuracy

New feature highlights

Static embeddings (same 2.3 GB fastText .vec: 884,790 words × 300 dims; Hazm loads it through gensim 4.4.0)

Rishe loads the same model 30× faster and pools sentences 2.5× faster, with bit-identical word vectors. Word lookup is at parity: both paths build a Python list, which dominates the cost.

The normalizer operators are 1.5–6× faster than Shekar's per operator (the exception is ya, 0.7×) with exact output agreement of 100% on 8 of 11 operators and 99.04% on verbal spacing. normalize_stages is opt-in; the default normalize path is unchanged.

The full core tables, batch scaling and method are in docs/BENCHMARKS.md. The feature tables, proxy metrics and caveats are in docs/FEATURE_BENCHMARKS.md.

Reproduce:

python scripts/benchmark.py             # core token/lemma/speed
python scripts/benchmark_features.py   # stemming, conjugation, splitting, spelling
python scripts/benchmark_normalizer.py # 11 normalizer operators vs Shekar/Hazm
python scripts/benchmark_maskers.py    # 12 maskers vs Shekar
python scripts/benchmark_embeddings.py # static embeddings vs Hazm/gensim

Install

pip install rishe        # Python >= 3.9

or, from Rust (Cargo.toml): rishe = "0.4".

The base install has no Python dependencies and no model downloads. Optional models are pulled explicitly and nothing touches the network at runtime.

Usage

from rishe import Pipeline

pipe = Pipeline()                         # unify digits, strip diacritics, fix ZWNJ

doc = pipe("کتاب‌های خوب را به دوستانم می‌دهم.")
doc.normalized_text, doc.formal_text, doc.sentences
doc.tokens, doc.lemmas, doc.ezafe_markers

docs = pipe.pipe(texts)                   # parallel batch -> list[ProcessedDoc]

pipe.normalize("مي كنم 2026")             # 'می‌کنم ۲۰۲۶'
pipe.tokenize("سلام، دنیا!")              # raw tokenization, no normalization
pipe.word_tokenize("سلام، دنیا!")         # normalize + tokenize
pipe.sent_tokenize(text)                  # sentence splitting only
pipe.lemmatize("می‌روم")                   # 'رفت'  (past-stem lemmas, UD style)
pipe.lemma_full("می‌روم")                 # ('رفت', 'رو')  (past + present stem)

pipe(text) also rewrites colloquial words to formal ones, so its tokens can differ from the surface text; word_tokenize tokenizes without rewriting. tokenize is the fast raw path used for the tokenization benchmarks.

Morphology and utilities

pipe.stem("کتاب‌ها")                      # 'کتاب'
pipe.conjugate("رفت")["present"][:3]      # ['می‌روم', 'می‌روی', 'می‌رود']
pipe.comparative("بزرگ")                  # 'بزرگ‌تر'
pipe.superlative("بزرگ")                  # 'بزرگ‌ترین'
pipe.plural("کتاب")                       # 'کتاب‌ها'
pipe.possessive("خانه", 0)                # 'خانه‌ام'
pipe.split_token("بهش")                   # ['به', 'او']
pipe.spell_suggest("سسلام")               # ['سلام', 'اسلام', ...]
pipe.spell_correct("سسلام بر ششما")       # 'سلام بر شما'
pipe.tokenize_spans("سلام دنیا")          # [('سلام', 0, 4), ('دنیا', 5, 9)]  (char offsets)
pipe.remove_stopwords(tokens)             # drop Persian stopwords
pipe.process_stages(text, ["normalize", "sentences"])  # choose pipeline stages

Normalizer operators

normalize_stages applies the P1 normalizer operators in the order you list them: alphabet, arabic_unicode, ya, ya_joda, digits, punctuation, repeated_letters, orthographic_spacing, punctuation_spacing, word_spacing, verbal_spacing, and spacing (all four spacing operators).

pipe.normalize_stages("﷽ 1𝟮3٤٥⓺", ["arabic_unicode", "digits"])
# 'بسم الله الرحمن الرحیم ۱۲۳۴۵۶'

pipe.normalize_stages("سلام(دنیا) ! می روم", ["punctuation_spacing", "verbal_spacing"])
# 'سلام (دنیا)! می‌روم'

word_spacing and verbal_spacing validate against the compiled lexicons, so they only join forms the engine knows (کتاب ها → کتاب‌ها, رفته ام → رفته‌ام). Unknown stage names raise ValueError.

Masking

mask_stages removes or masks URLs, emails, mentions, hashtags, emoji, HTML tags, non-Persian letters, stopwords, offensive words, digits, diacritics and punctuation. Stage names are url, email, mention, hashtag, emoji, html, non_persian, stopwords, offensive, digits, diacritics and punctuation.

pipe.mask_stages("سلام @user، به https://rishe.dev یا info@rishe.dev بنویس.",
                 ["url", "email", "mention"])
# 'سلام   به <URL> یا <EMAIL> بنویس.'

pipe.mask_stages("این کتاب است", ["stopwords"], mask_token="[MASK]")
# '[MASK] کتاب [MASK]'

Defaults are <URL> for URLs, <EMAIL> for emails, a space for mentions, hashtags and HTML tags, and removal for everything else. Pass mask_token to override all stages. Masking keeps the spaces left behind; run normalize_stages(..., ["spacing"]) afterwards to collapse them. Unknown stage names raise ValueError.

Spell correction uses a 190k-word frequency lexicon and edit-distance 1; typos with two or more edits are left unchanged.

Utilities

pipe.number_to_words(1404)                    # 'یک هزار و چهارصد و چهار'
pipe.number_to_words("3.14")                  # 'سه ممیز چهارده'
pipe.ngrams("این یک متن", 1, 2)                # ['این', 'یک', 'متن', 'این یک', 'یک متن']
pipe.persianize("فایل را با کامپیوتر باز کن")   # 'پرونده را با رایانه باز کن'
pipe.persianize_suggest("فایل")               # [('فایل', ['پرونده'], 0, 4)]
flatten(["الف", ["ب", "پ"]])                  # 'الف', 'ب', 'پ'

persianize uses a built-in curated loanword map; persianize_suggest returns the alternatives and character spans.

Augmentation

keyboard_noise, ocr_noise and whitespace_noise simulate typing, OCR and spacing errors. Pass seed= for reproducible output.

pipe.keyboard_noise(text, seed=42)
pipe.ocr_noise(text, seed=42)
pipe.whitespace_noise(text, seed=42)

Keywords and transliteration

pipe.rake(text, top_n=5)
pipe.textrank(text, top_n=5)
pipe.to_tajik("خانه")      # 'хона'
pipe.to_persian("хона")    # 'خانه'

Transliteration is a rule-based script mapping; it preserves the consonant skeleton and is not a phonetic translation.

Tokenizer options

pipe.word_tokenize_with("رفته است", join_verb_parts=True)   # ['رفته_است']
pipe.word_tokenize_with("info@a.co", replace_emails=True)   # ['EMAIL']

Options: join_verb_parts, join_abbreviations, separate_emoji, replace_links, replace_ids, replace_emails, replace_numbers, replace_hashtags. All are off by default, so plain word_tokenize is unchanged.

Embeddings

Static fastText/word2vec vectors load from a .vec file into the Rust core and stay there: lookups and sentence pooling run without Python objects, numpy or gensim, and the model is shared read-only across threads. Downloads are explicit and checksum-verified:

rishe hub pull hazm-fasttext-300      # 2.3 GB, cached in ~/.cache/rishe/models

from rishe import Embeddings, hub

emb = Embeddings(hub.path("hazm-fasttext-300"))
emb.dim, len(emb)                     # vector size, vocabulary size
emb.vector("کتاب")                    # list[float] or None
emb.similarity("کتاب", "دفتر")         # cosine or None
emb.sentence_vector("این کتاب خوب است") # mean of in-vocab token vectors
emb.sentence_vectors(["یک متن", "متن دیگر"])  # parallel, GIL released
emb.most_similar("کتاب", 5)           # [(word, cosine), ...]

Out-of-vocabulary tokens are skipped in sentence vectors; vector() and similarity() return None. RISHE_HOME overrides the cache root, and rishe.hub.pull(url, sha256=...) pulls a custom model. Only .vec text files are supported: fastText .bin and GloVe files are not.

Model packs

POS tagging, NER and ezafe detection ship as three compact task packs. Each pack is a separate, checksum-verified download (under 50 MB; the model is 45 MB and the tokenizer adds 2.4 MB); the base install stays free of Python ML dependencies. The packs run on the pure-Rust runtime inside Rishe, so there is no ONNX Runtime install and no Python tokenizer stack.

rishe hub pull pos-compact       # also: ner-compact, ezafe-compact

from rishe import ModelPack, hub

pos = ModelPack(hub.path_pack("pos-compact"))
pos.tag("کتاب‌های خوب را به دوستانم می‌دهم.")
# [('کتاب‌های', 'NOUN'), ('خوب', 'ADJ'), ('را', 'ADP'), ('به', 'ADP'),
#  ('دوستانم', 'NOUN'), ('می‌دهم', 'VERB'), ('.', 'PUNCT')]

pos.tag_words(["کتاب‌های", "خوب"])   # pre-tokenized input

ner = ModelPack(hub.pull_pack("ner-compact"))
ner.tag("سرلشگر فیروزآبادی به پادشاه عربستان گفت.")
# [..., ('فیروزآبادی', 'B_PER'), ..., ('عربستان', 'B_LOC'), ...]

ezafe = ModelPack(hub.pull_pack("ezafe-compact"))
ezafe.tag_words(["کتابخانه", "مرکزی", "دانشگاه", "شریف"])
# [('کتابخانه', 'EZAFE'), ('مرکزی', 'EZAFE'), ('دانشگاه', 'EZAFE'),
#  ('شریف', 'NO_EZAFE')]

ModelPack releases the GIL during inference. hub.pull_pack(name) downloads and returns the directory; hub.path_pack(name) returns an existing cache entry and raises if it is not there. From Rust, enable the models feature (this optional feature needs Rust 1.91; the default crate still builds on 1.75) and call rishe::models::ModelPack::load(path)?.tag_text(text)?.

Corpora

from rishe import ConlluReader, PeymaReader, PlainTextReader

for sentence in ConlluReader("data/corpora/fa_perdt-ud-test.conllu"):
    print(sentence.text, sentence.tokens[0].lemma)

ConlluReader parses UD/CoNLL-U, PeymaReader parses the PEYMA test format, and PlainTextReader reads one sentence per line. Other corpus-specific formats can be added as needed.

CLI

rishe normalize "مي كنم"
rishe tokenize "سلام دنیا" --json
rishe mask "info@a.co" --stages email
printf '%s\n' '{"command":"tokenize","text":"سلام دنیا"}' | rishe serve

Convenience classes

Module-level helpers have object-style counterparts when you want a long-lived instance instead of building one per call:

from rishe import Lemmatizer, Normalizer, WordTokenizer, Stemmer

normalizer = Normalizer()
normalizer.normalize("مي كنم")        # 'می‌کنم'

lemmatizer = Lemmatizer()
lemmatizer.lemmatize("می‌روم")         # 'رفت#رو'  (past#present)

WordTokenizer().tokenize("رفته است")   # ['رفته_است']

Normalizer, WordTokenizer, SentenceTokenizer, Lemmatizer, Stemmer and TokenSplitter are all available this way, plus the module-level word_tokenize and sent_tokenize. They take no lexicon paths: Rishe compiles its lexicons in.

Rust

use rishe::Pipeline;

let pipe = Pipeline::create(true, true, true);
let doc = pipe.process("کتاب‌های خوب را به دوستانم می‌دهم.");
println!("{:?}", doc.tokens);
println!("{}", pipe.stem("کتاب‌ها"));

Scope

Rishe is a text-preprocessing and morphology engine: normalization, tokenization, sentence splitting, lemmatization, stemming, verb conjugation, inflection, token splitting, spell correction, stopword removal, ezafe detection, informal → formal conversion and static word/sentence embeddings.

Optional model packs add POS tagging, NER and a learned ezafe tagger. They are separate downloads from the Hugging Face Hub, so the base install stays zero-dependency and fast. The 0.5.0 engine loads the packs directly with ModelPack; no ONNX Runtime install is needed.

Dependency parsing, chunking, contextual embeddings and text classification are not in the base yet.

Building from source

cargo test
maturin develop --release     # Python extension into the current venv
python scripts/benchmark.py   # needs: hazm shekar conllu
python scripts/benchmark_features.py

scripts/extract_lexicons.py regenerates data/*.tsv and data/word_freq.tsv from Hazm's data files; build.rs compiles them into FSTs.

License

Dual-licensed under Apache-2.0 or MIT, at your option. The bundled lexicons, stopwords and word frequencies are derived from Hazm data (MIT); see NOTICE.md.