Skip to main content

persian-text-guard

Persian text normalization and evasion-resistant profanity filtering for Python.

فارسی · Full project README · .NET package · npm package

A word list is easy to get around. Type an Arabic ي instead of ی, slip a zero-width character inside a word, write f u c k, sh1t, کیییییر or k0s, and a plain word in text check sees nothing. persian-text-guard reads through those tricks, and just as importantly, it doesn't flag ordinary messages: هر کس, تخم مرغ, class, push it and Scunthorpe all pass.

It is the Python port of PersianTextGuard, and gives the same answers as the .NET and JavaScript packages for every case in the shared conformance corpus.

  • Pure Python, with full type hints (py.typed), for CPython 3.11 and later, free-threaded 3.14 included.
  • No dependencies.
  • Safe to share across threads.

Installation

pip install persian-text-guard

or, in a project managed by uv:

uv add persian-text-guard

Quick start

from persian_text_guard import ProfanityFilter, WordList

# Build once and share it: a filter never changes after it is built.
filter = ProfanityFilter(WordList.persian_default())

assert not filter.contains_profanity("سلام، سفارشم کی میرسه؟")
assert filter.contains_profanity("ک.ی.ر")
assert filter.contains_profanity("f u c k")

What matched, and why

from persian_text_guard import EvasionKind, ProfanityFilter, WordCategory, WordList

filter = ProfanityFilter(WordList.persian_default())
match = filter.find_match("sh1iiit")

assert match is not None
assert match.word.text == "shit"
assert match.word.category == WordCategory.PROFANITY
assert match.evasion == (EvasionKind.REPEATED_LETTERS, EvasionKind.LOOKALIKE_CHARACTERS)
assert filter.find_match("hello") is None

Every match, with its position

Positions are Python string indexes (code points), so slicing gives the matched text:

from persian_text_guard import ProfanityFilter, WordList

filter = ProfanityFilter(WordList.persian_default())
message = "sh1t and f u c k"

found = [
    (match.word.text, match.index, match.length, message[match.index : match.index + match.length])
    for match in filter.find_matches(message)
]
assert found == [("shit", 0, 4, "sh1t"), ("fuck", 9, 7, "f u c k")]

This holds for emoji and every other character outside the Basic Multilingual Plane too: each counts as one position, as it does everywhere else in Python.

from persian_text_guard import ProfanityFilter, WordList

filter = ProfanityFilter(WordList.persian_default())
message = "\N{GRINNING FACE} کیر"

match = filter.find_match(message)
assert match is not None
assert (match.index, match.length) == (2, 3)
assert message[match.index : match.index + match.length] == "کیر"

Censoring

from persian_text_guard import ProfanityFilter, WordList

filter = ProfanityFilter(WordList.persian_default())

assert filter.censor("kir and motherfucker") == "**** and ****"
assert filter.censor("جنده\N{ZERO WIDTH NON-JOINER}ها رو ببین") == "**** رو ببین"
assert filter.censor("this is kir", "#") == "this is ####"
assert filter.censor("hello") == "hello"

Whole words are hidden, every mask is four characters, and the output is always clean. Clean text comes back unchanged, and everything outside a hidden word is returned exactly as given. The mask can be any symbol or punctuation; a letter, digit, whitespace or control character raises ValueError.

Categories

Every bundled entry has a category: profanity, sexual, insult, slur, harassment or mild.

from persian_text_guard import ProfanityFilter, WordCategory, WordList

everyday = ProfanityFilter(WordList.persian_default())  # everything except mild
strict = ProfanityFilter(WordList.all())  # mild too
slurs_and_abuse = ProfanityFilter(WordList.bundled(WordCategory.SLUR, "harassment"))

assert not everyday.contains_profanity("این فیلم آشغال بود")
assert strict.contains_profanity("این فیلم آشغال بود")
assert slurs_and_abuse.contains_profanity("kys")
assert not slurs_and_abuse.contains_profanity("کیر")

A category, like a match mode or a normalization step, can be given as an enum member or as its name ("slur"); a type checker rejects a name that does not exist, and so does the code at run time.

Your own words

from persian_text_guard import BannedWord, ProfanityFilter, WordList, WordMatchMode

filter = ProfanityFilter(
    [
        BannedWord("اسپم"),  # whole word (the default)
        BannedWord("casino", WordMatchMode.ANYWHERE),  # inside longer words too
    ]
)
assert filter.contains_profanity("این پیام اسپم است")
assert filter.contains_profanity("onlinecasino.example")

# Or combine your entries with the bundled list:
combined = ProfanityFilter([*WordList.persian_default(), BannedWord("casino", "anywhere", "mild")])
assert combined.contains_profanity("casino night")

Word-list files

Word lists can also be kept in a text file, one entry per line. WordList.load reads a path or an open file as UTF-8, and WordList.parse reads text you already have:

import tempfile
from pathlib import Path

from persian_text_guard import ProfanityFilter, WordCategory, WordList, WordMatchMode

text = "# comments start with #\n[insult]\nidiot\n~scam\n"

words = WordList.parse(text)
assert [(w.text, w.mode, w.category) for w in words] == [
    ("idiot", WordMatchMode.WHOLE_WORD, WordCategory.INSULT),
    ("scam", WordMatchMode.ANYWHERE, WordCategory.INSULT),
]
assert ProfanityFilter(words).contains_profanity("scammers everywhere")

with tempfile.TemporaryDirectory() as folder:
    path = Path(folder) / "words.txt"
    path.write_text(text, encoding="utf-8")
    assert WordList.load(path) == words
    with path.open("rb") as file:
        assert WordList.load(file) == words  # an open file is read, and left open

A heading such as [insult] names a category in any letter case. An unknown heading raises a WordListFormatError, a ValueError with the heading's line. A UTF-8 byte order mark is ignored.

Options

Every evasion the filter reads through can be turned off:

from persian_text_guard import ProfanityFilter, ProfanityFilterOptions, WordList

literal = ProfanityFilter(WordList.persian_default(), ProfanityFilterOptions(join_spaced_letters=False))

assert not literal.contains_profanity("f u c k")
assert literal.contains_profanity("fuuuck")

Normalization

from persian_text_guard import NormalizationStep, normalize, to_ascii_digits, to_persian_digits, tokenize

text = "كتاب\N{ZERO WIDTH NON-JOINER}هاي  ۱۲ ABC"

# "comparison" (the default): everything folded, for searching and matching.
assert normalize(text) == "کتابهای 12 abc"
# "standard": safe to store and show; keeps the zero-width non-joiner, Persian digits and case.
assert normalize(text, "standard") == "کتاب\N{ZERO WIDTH NON-JOINER}های ۱۲ ABC"
# Or any combination of steps, as members or names.
assert normalize("ABC ي", [NormalizationStep.LOWER_CASE, "unifyLetters"]) == "abc ی"

assert tokenize("سلام، دنیا! خوبی؟") == ["سلام", "دنیا", "خوبی"]
assert to_persian_digits("2 ساعت پیش") == "۲ ساعت پیش"
assert to_ascii_digits("۱۴۰۴/۰۵/۱۴") == "1404/05/14"

Thread safety

A filter is immutable after it is built, and the bundled lists are parsed once, under a lock. Build one filter and share it between threads, including on free-threaded CPython 3.14 with the GIL disabled:

from concurrent.futures import ThreadPoolExecutor

from persian_text_guard import ProfanityFilter, WordList

filter = ProfanityFilter(WordList.persian_default())
messages = ["سلام", "ک.ی.ر", "f u c k", "hello"] * 50

with ThreadPoolExecutor(max_workers=8) as pool:
    results = list(pool.map(filter.contains_profanity, messages))

assert results == [filter.contains_profanity(message) for message in messages]

Validating input

Checking, finding matches, censoring, normalizing and tokenizing accept any str, including empty text, broken emoji, lone surrogates and very long messages, and treat None as an empty message. They never raise for text.

Anything else, such as bytes, a number or a list, raises TypeError, so a missing check shows up at once instead of an unchecked value passing as clean. Values from a request are untrusted, so check that the message is a string first:

import json

from persian_text_guard import ProfanityFilter, WordList

filter = ProfanityFilter(WordList.persian_default())
body = json.loads('{"message": ["kir"]}')

message = body.get("message")
if not isinstance(message, str):
    outcome = "rejected: message must be text"
else:
    outcome = filter.censor(message)
assert outcome == "rejected: message must be text"
Error Raised by When
TypeError every text function, censor, WordList.parse, the constructors A value that is not a str (or None, where None is allowed), an option that is not a bool, or an entry that is not a BannedWord
ValueError censor, enum parameters, normalize An invalid mask, or an unknown category, mode, step or preset name
WordListFormatError (a ValueError) WordList.parse, WordList.load An unknown section heading; .line is 1-based
OSError, UnicodeDecodeError WordList.load The file cannot be read, or is not UTF-8

.NET, JavaScript and Python

.NET JavaScript/TypeScript Python
new ProfanityFilter(words, options) new ProfanityFilter(words, options) ProfanityFilter(words, options)
filter.Count filter.count filter.count
filter.ContainsProfanity(text) filter.containsProfanity(text) filter.contains_profanity(text)
filter.FindMatch(text) filter.findMatch(text) filter.find_match(text) (None when there is no match)
filter.FindMatches(text) filter.findMatches(text) filter.find_matches(text) (a tuple)
filter.Censor(text, '#') filter.censor(text, '#') filter.censor(text, "#")
new BannedWord("x", WordMatchMode.Anywhere) { Category = WordCategory.Slur } { text: 'x', mode: 'anywhere', category: 'slur' } BannedWord("x", WordMatchMode.ANYWHERE, WordCategory.SLUR)
ProfanityFilterOptions { SqueezeRepeatedLetters = false } { squeezeRepeatedLetters: false } ProfanityFilterOptions(squeeze_repeated_letters=False)
match.Word, match.Evasion, match.Index, match.Length match.word, .evasion, .index, .length match.word, .evasion, .index, .length (code points)
EvasionKind.LookalikeCharacters | EvasionKind.RepeatedLetters ['repeatedLetters', 'lookalikeCharacters'] (EvasionKind.REPEATED_LETTERS, EvasionKind.LOOKALIKE_CHARACTERS)
WordList.All, WordList.PersianDefault WordList.all, WordList.persianDefault WordList.all(), WordList.persian_default()
WordList.Bundled(WordCategory.Slur) WordList.bundled('slur') WordList.bundled(WordCategory.SLUR) or WordList.bundled("slur")
WordList.Parse(text) WordList.parse(text) WordList.parse(text)
WordList.Load(stream) read the file, then WordList.parse(text) WordList.load(path_or_file)
PersianNormalizer.Normalize(text, PersianNormalization.Standard) normalize(text, 'standard') normalize(text, "standard")
PersianNormalizer.Tokenize(text) tokenize(text) tokenize(text)
PersianNormalizer.ToPersianDigits(text) toPersianDigits(text) to_persian_digits(text)
FormatException WordListFormatError (.line) WordListFormatError (.line), a ValueError
ArgumentException (mask) RangeError ValueError

Performance

Against WordList.persian_default() (about 1,000 entries), on CPython 3.14.6 (Intel Core i7-9700K, pyperf with --rigorous):

Operation Mean Operations/s
Short clean message (5 words) 91.4 µs 10,937
Long clean message (60 words) 909 µs 1,100
Message with evasions 50.4 µs 19,850
Normalize a long message 35.1 µs 28,454
Build a filter from the bundled list 13.3 ms 75
find_matches, clean short message 90.4 µs 11,059
find_matches, message with three banned words 240 µs 4,169
censor, short message with one banned word 118 µs 8,461
censor, 60-word message with three banned words 3.2 ms 311
A 132,000-character message 210 ms 5

Build the filter once and reuse it. The messages are the same as the .NET and JavaScript benchmarks'; run them with uv run python bench/bench_filter.py -o .bench/results.json in python/, and print the table with uv run python scripts/bench_table.py .bench/results.json.

Limitations

  • Unicode versions. The filter uses the running Python's own Unicode data (unicodedata), through a layer that reproduces .NET's rules for whitespace, case and categories. CPython 3.14 and .NET 10 agree on the category of every code point. Older versions differ only on characters assigned in Unicode 15 or later, and Python 3.11 also does not fold the 62 Cyrillic modifier letters U+1E030–U+1E06D in compatibility normalization. The conformance corpus contains none of these characters; if a difference matters to you, please open an issue.
  • Values that are not text. bytes, numbers and other objects raise TypeError instead of being read as an empty or converted message; see Validating input.
  • Interpreters. CPython 3.11 to 3.14, including free-threaded 3.14, is tested. Other interpreters, such as PyPy, may work but are not tested.
  • The evasions the filter reads through, and what it deliberately does not catch, are listed in the project README.

فارسی

این بسته نسخهٔ پایتونِ پروژهٔ PersianTextGuard است: یکسان‌سازی متن فارسی و فیلتر کلمات نامناسب که ترفندهای دور زدن فهرست کلمات را می‌شناسد، مثل «ک.ی.ر»، حروف فاصله‌دار، حروف عربی به‌جای فارسی و نویسه‌های نامرئی، و در عین حال پیام‌های عادی مثل «هر کس» و «تخم مرغ» را رد نمی‌کند.

برای هر پیام همان پاسخی را می‌دهد که بسته‌های دات‌نت و جاوااسکریپت می‌دهند، و جایگاه‌ها را مثل خودِ پایتون می‌شمارد.

نصب

pip install persian-text-guard

شروع سریع

فیلتر را یک بار بسازید و همه‌جا از همان استفاده کنید؛ فیلتر بعد از ساخته شدن هرگز تغییر نمی‌کند و استفادهٔ هم‌زمان از چند نخ هم امن است.

from persian_text_guard import ProfanityFilter, WordList

# فیلتر را یک بار بسازید و به اشتراک بگذارید.
# Build once and share it: a filter never changes after it is built.
filter = ProfanityFilter(WordList.persian_default())

assert not filter.contains_profanity("سلام، سفارشم کی میرسه؟")
assert filter.contains_profanity("ک.ی.ر")

سانسور

هر کلمهٔ نامناسب، با پسوندها و فاصله‌هایش، با چهار نویسهٔ ثابت پوشانده می‌شود و بقیهٔ متن دست نمی‌خورد.

from persian_text_guard import ProfanityFilter, WordList

# کلمات نامناسب با چهار نویسه پوشانده می‌شوند.
# Every banned word is hidden behind a four-character mask.
filter = ProfanityFilter(WordList.persian_default())

assert filter.censor("جنده\N{ZERO WIDTH NON-JOINER}ها رو ببین") == "**** رو ببین"
assert filter.censor("this is kir", "#") == "this is ####"

بررسی ورودی

هر رشته‌ای پذیرفته می‌شود و مقدار خالیِ پایتون مثل پیام خالی است، اما هر مقدار دیگری، مثل عدد یا بایت یا فهرست، خطای نوع می‌دهد. پیش از فیلتر کردنِ داده‌های درخواست، بررسی کنید که پیام از نوع رشته باشد.

مستندات کامل، فهرست ترفندهایی که شناخته می‌شوند و محدودیت‌ها در راهنمای اصلی پروژه آمده است.

Metadata

Release files for persian-text-guard 1.4.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for persian-text-guard 1.4.0
File Size Uploaded Provenance Actions
persian_text_guard-1.4.0.tar.gz 55.2 kB With provenance Details for persian_text_guard-1.4.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for persian-text-guard 1.4.0
File Interpreter ABI Platform Actions
persian_text_guard-1.4.0-py3-none-any.whl Python 3 none any Details for persian_text_guard-1.4.0-py3-none-any.whl

Total release size: 111.5 kB

Release files / persian_text_guard-1.4.0.tar.gz

Download URL persian_text_guard-1.4.0.tar.gz
Size 55.2 kB
Tags Source
SHA-256 checksum
How to use checksums
e5017bd4dacc49a629d6228d6771d0c99b1892a606e63ecf1e1fdc107de86f06
BLAKE2b-256 checksum
How to use checksums
27587ab25930d755d9d2b61c4a64f205384d506da68282b44cf615dae45e168a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 19, 2026.

Transparency log

Release files / persian_text_guard-1.4.0-py3-none-any.whl

Download URL persian_text_guard-1.4.0-py3-none-any.whl
Size 56.3 kB
Tags Python 3
SHA-256 checksum
How to use checksums
d0577af458ce406e0f60ac69083814a20ddb1613a42ff9fe6cbb21efa95f4b7d
BLAKE2b-256 checksum
How to use checksums
252eadb4ddf4a19b1a15371c9e69ca8bbbf6629ad06532e8004cdae02b7d7bca
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 19, 2026.

Transparency log

Security for persian-text-guard 1.4.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

Both files in this release have been uploaded with provenance metadata.

Select from 2 files

Attestation for persian_text_guard-1.4.0.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Sep 19, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.4.0 This release

2 release files Full provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page