prompt-canon — Canonicalize Before You Guard: a Unicode Prompt-Injection Normalizer for LLMs

prompt-canon is a small, pure-standard-library Python library that canonicalizes text before you guard it. It is a unicode prompt injection normalizer for LLM pipelines: it strips invisible characters, maps cross-script homoglyphs to an ASCII skeleton, and applies a locale-independent case fold — so that your guardrail, classifier, moderation model, or allow/deny list sees one canonical form instead of the many visually identical variants an attacker can spell.

Canonicalize before you guard. A detector that never sees the real bytes cannot block them. "İGNORE previous instructions" spelled with a Turkish dotted capital I, a zero-width space, and a Cyrillic look-alike is invisible to a naive keyword or embedding filter — until you normalize it first.

Keywords: unicode prompt injection normalizer, homoglyph filter for LLM, invisible unicode remover, zero-width character stripper, prompt injection defense, OWASP LLM Top 10, MITRE ATLAS, Unicode confusables, canonicalization.

• Zero runtime dependencies — only unicodedata and re from the standard library.
• Python 3.8+.
• Deterministic and offline — no model, no network, no telemetry.
• Transparent — every character it removes, rewrites, or flags is reported as a structured finding.

---

Install

pip install prompt-canon

Or from source:

git clone https://github.com/fevziegeyurtsevenler/prompt-canon
cd prompt-canon
pip install -e ".[test]"

30-second quickstart

from prompt_canon import canonicalize

# A prompt-injection string using a Turkish dotted-I, a Cyrillic 'р',
# and an embedded zero-width space.
raw = "İGNORE​ рrevious instructions"

result = canonicalize(raw, fold_case=True)

print(result.text)
# -> "ignore previous instructions"

for f in result.findings:
    print(f["kind"], f["codepoint"], f["note"])
# zero-width  U+200B  removed; zero-width / invisible character
# confusable  U+0440  mapped 'р' -> 'p' (CYRILLIC SMALL LETTER ER)

Now hand result.text — not raw — to your guardrail, regex, moderation API, or classifier.

Individual transforms

from prompt_canon import strip_invisible, map_confusables, fold_case

strip_invisible("ig​nore")     # "ignore"  (zero-width space removed)
map_confusables("аdmin")            # "admin"   (Cyrillic 'а' -> 'a')
fold_case("İGNORE")                 # "ignore"  (Turkish dotted I handled)
fold_case("straße")                 # "strasse" (German sharp S handled)

What it does

Transform	Behaviour
Zero-width / invisible	Removes U+200B, U+2060–U+2064, and non-BOM U+FEFF.
Tag block	Removes U+E0000–U+E007F (the "ASCII smuggling" tag characters).
Bidi controls	Removes and flags U+202A–U+202E and U+2066–U+2069 (Trojan Source, CVE-2021-42574).
ZWJ / ZWNJ	Keeps and flags U+200D / U+200C — they are legitimate in emoji and Persian/Arabic/Indic text.
Byte-order mark	A leading U+FEFF is preserved; inner U+FEFF is stripped.
Confusables	Maps curated Cyrillic / Greek / Latin-Extended / fullwidth homoglyphs to their ASCII skeleton.
Case fold	Locale-independent fold that also fixes Turkish dotted/dotless I and German ß. Off by default.

Every removed, rewritten, or flagged character is returned as a finding:

{"kind": "bidi", "codepoint": "U+202E", "offset": 4,
 "note": "removed; bidirectional control (Trojan Source / CVE-2021-42574 risk)"}

Why NFKC and str.casefold() are not enough

The reflex is to reach for unicodedata.normalize("NFKC", text) and str.casefold(). Both are useful, and prompt-canon is designed to sit alongside them — but neither closes the gap on their own.

The İGNORE worked example

Take the classic injection trigger ignore, spelled with a Turkish dotted capital I (İ, U+0130):

>>> "İGNORE".casefold()
'i̇gnore'          # note the extra combining dot: 'i' + U+0307
>>> "İGNORE".casefold() == "ignore"
False

str.casefold() is deliberately locale-independent, so it folds U+0130 to i plus a combining dot above — not to a bare ASCII i. A keyword or regex check for "ignore" therefore misses it. The dotless variant is just as bad:

>>> "ıgnore".casefold()       # Turkish dotless small i, U+0131
'ıgnore'                      # unchanged — still not "ignore"

And NFKC does not fix it either, because these are distinct, "normal" letters, not compatibility characters:

>>> import unicodedata
>>> unicodedata.normalize("NFKC", "İGNORE").casefold() == "ignore"
False

prompt-canon's fold_case normalizes both Turkish I forms to ASCII i, applies casefold (which already turns ß/ẞ into ss), and drops the residual combining dot:

>>> from prompt_canon import fold_case
>>> fold_case("İGNORE")
'ignore'
>>> fold_case("ıgnore")
'ignore'
>>> fold_case("straße")
'strasse'

Similarly, NFKC leaves whole classes of attack untouched: it does not remove zero-width or bidi controls, and it does not collapse cross-script homoglyphs — a Cyrillic а (U+0430) stays a Cyrillic а under NFKC. Those are exactly the gaps strip_invisible and map_confusables are built to cover.

Where this fits: OWASP LLM Top 10 and MITRE ATLAS

prompt-canon is authorized, defensive security-testing and hardening tooling. It maps to:

• OWASP Top 10 for LLM Applications — LLM01: Prompt Injection. Unicode obfuscation (invisible characters, homoglyphs, bidi reordering) is a common way to smuggle injection payloads past filters. Canonicalizing first shrinks that evasion surface.
• MITRE ATLAS. Relevant to adversarial techniques around LLM Prompt Injection and evasion/obfuscation of input-side defenses. Use prompt-canon as a normalization control in front of your detection layer.

Responsible use

This library is a defensive normalization layer. It is meant to be run on untrusted input before your guardrails, to make evasion harder and to give you an auditable record (the findings) of what was cleaned. It does not generate attacks. Do not use the findings output or the confusable table to build or tune payloads against systems you are not authorized to test.

Honesty: what prompt-canon is and is not

Keeping this honest matters more than keeping it impressive.

• It is a normalization / coverage layer, not a detector. prompt-canon never decides whether text is malicious. It reduces many visually equivalent spellings to one canonical form so that your detector has a fair shot. Pair it with a real classifier or policy engine.
• The confusable table is curated, not exhaustive. It covers common Cyrillic, Greek, Latin-Extended, and fullwidth look-alikes — not the entire Unicode confusables database (UTS #39). It will miss glyphs it has never seen.
• Normalization can cause false positives. Mapping homoglyphs and folding case is lossy by design. Legitimate multilingual text — a genuinely Greek or Cyrillic word, a name, a URL — can be altered. Canonicalize the copy you feed to your detector; do not overwrite the text you store or display to the user without thought. Case folding is off by default precisely so that a clean ASCII input is returned byte-for-byte unchanged.
• Idempotent, but destructive. canonicalize(canonicalize(x)) == canonicalize(x), and the transforms discard information on purpose.

Prior art and complementary tools

prompt-canon is a focused building block, not a replacement for these projects — use it with them:

• promptfoo — red-teaming and eval framework that includes ASCII-smuggling / invisible-Unicode strategies for testing models. prompt-canon is the normalization side of that same problem.
• LLM Guard (Protect AI) — a broader input/output scanner suite (prompt-injection, PII, toxicity, and more).
• Microsoft Presidio — PII detection and anonymization, which also benefits from canonical input.
• The Unicode Security Mechanisms standard, UTS #39, is the authoritative source for the confusable equivalences this library draws a curated subset from.

If you need full detection, orchestration, or PII handling, reach for those. prompt-canon does one thing: it canonicalizes the bytes first.

Cross-link: unicode-threat-reveal

prompt-canon is the dependency behind the unicode-threat-reveal Hugging Face Space, which visualizes what this library removes and flags — highlighting the invisible characters, homoglyphs, and bidi controls hiding inside a pasted prompt. This library is the engine; that Space is the interactive lens on top of it.

Development

pip install -e ".[test]"
python -m pytest -q

CI runs the suite on Python 3.9–3.12 (see .github/workflows/test.yml).

License

Apache License 2.0 — see LICENSE and NOTICE.