unidash

Fold Unicode dashes and punctuation to plain ASCII while leaving letters untouched.

Text pasted from word processors, PDFs, CJK input methods and web pages is full of characters that look like - but are not: en and em dashes, the minus sign, the non-breaking hyphen, fullwidth and small forms, and the invisible soft hyphen. They break search, diffs, command lines and parsers. unidash knows every character with the Unicode Dash property, plus the soft hyphen, and folds them the way you choose.

Install

pip install unidash

Python 3.10 or newer, no dependencies.

Dashes

fold_dashes(text, form="ascii") has three forms. ascii turns every dash into one - (wavy dashes into ~). tex keeps the en and em dash apart the way TeX and Markdown do, as -- and ---. typographic keeps the real dashes and only replaces the small, vertical and fullwidth compatibility forms with their standard counterparts.

from unidash import fold_dashes

text = "Pages 10\u201320 \u2014 see \u22125\uff0d6"
print(fold_dashes(text))
print(fold_dashes(text, "tex"))
print(ascii(fold_dashes("a\ufe58b\uff0dc", "typographic")))
# Pages 10-20 - see -5-6
# Pages 10--20 --- see -5-6
# 'a\u2014b-c'

A minus sign is always a single - in the ascii and tex forms (typographic keeps U+2212), so negative numbers stay negative, and characters that are already ASCII are never touched: 1-2 and a hand-written -- come out as they went in.

Soft hyphens, non-breaking hyphens and script hyphens

The soft hyphen (U+00AD) is a hint for where a word may break, not a character, so it is removed unless you pass keep_soft_hyphens=True. The non-breaking hyphen (U+2011) becomes - in the ASCII forms and stays in typographic, where its promise matters.

Hyphens that belong to one script (Armenian, the Hebrew maqaf, Canadian syllabics, Mongolian, the katakana double hyphen, Garay and Yezidi) are left alone, like that script's letters, unless you pass script_hyphens=True.

from unidash import fold_dashes

print(fold_dashes("co\u00adop\u00aderate"))
print(ascii(fold_dashes("co\u00adop", keep_soft_hyphens=True)))
print(ascii(fold_dashes("\u05e2\u05be\u05d1")))
print(ascii(fold_dashes("\u05e2\u05be\u05d1", script_hyphens=True)))
# cooperate
# 'co\xadop'
# '\u05e2\u05be\u05d1'
# '\u05e2-\u05d1'

Finding dashes

find_dashes yields a DashHit (offset, line, column, character and Unicode name) for every character fold_dashes would change with the same options, which makes a lint or a pre-commit check a one-liner. Lines end at LF, CR or CRLF, columns count code points from 1, and a byte order mark at the start of the text takes no column. is_dash(char) tells whether a character has the Unicode Dash property and DASHES is that set. dash_table(...) returns the str.translate table, and iter_fold_dashes(chunks, ...) folds an iterable of strings lazily, for large inputs.

from unidash import find_dashes, is_dash, iter_fold_dashes

for hit in find_dashes("first line\nsecond\u2014line\u00ad"):
    print(hit.line, hit.column, hit.codepoint, hit.name)
print(is_dash("\u2212"), is_dash("\u00ad"))
print("".join(iter_fold_dashes(["a\u2013", "b"])))
# 2 7 U+2014 EM DASH
# 2 12 U+00AD SOFT HYPHEN
# True False
# a-b

All punctuation

to_ascii_punct(text) folds the rest of the typographic punctuation too: curly quotes become ' and ", the ellipsis becomes ..., non-breaking and other Unicode spaces become a normal space, bullets become *, fullwidth punctuation from CJK input (U+FF01 to U+FF5E, letters and digits excepted) becomes its ASCII character, and dashes follow the ascii form. Accented letters pass through.

from unidash import to_ascii_punct

print(to_ascii_punct("\u201cCaf\u00e9\u201d\u2014it\u2019s 100\u00a0%\u2026"))
# "Café"-it's 100 %...

Command line

unidash FILE... writes the folded text to stdout (- or no file reads stdin). It takes --form, --script-hyphens, --keep-soft-hyphens, and --punct for the full punctuation fold. --check lists what would change and exits 1, for pre-commit hooks. Input must be UTF-8; unreadable files exit 2.

$ unidash --check notes.md
notes.md:2:6: U+2014 EM DASH
notes.md:3:3: U+00AD SOFT HYPHEN
$ unidash --form tex notes.md
Draft
Done --- ship it
coop

What counts as a dash

The classes follow the Unicode 18.0 Dash property and General_Category=Pd, and the tests check them against the Unicode data files and the UTS #39 confusables list.

Script hyphens, folded only with script_hyphens=True (to -, or to U+2010 in typographic): U+058A ARMENIAN HYPHEN, U+05BE HEBREW PUNCTUATION MAQAF, U+1400 CANADIAN SYLLABICS HYPHEN, U+1806 MONGOLIAN TODO SOFT HYPHEN, U+30A0 KATAKANA-HIRAGANA DOUBLE HYPHEN, U+10D6E GARAY HYPHEN and U+10EAD YEZIDI HYPHENATION MARK.

The double hyphens (U+2E40, and U+1400 and U+30A0 among the script hyphens) become - although UTS #39 lists them as confusable with =: they are hyphens in use, and = would change the meaning of the text.

Never folded: U+002D HYPHEN-MINUS, and U+207B SUPERSCRIPT MINUS and U+208B SUBSCRIPT MINUS, which sit next to superscript and subscript digits that stay as they are. Characters that only look like a hyphen are not dashes and are left alone: letters in other scripts, the Arabic full stop, the hyphen bullet, U+02D7 MODIFIER LETTER MINUS SIGN (a tone mark), and minus-like symbols without the Dash property such as U+2052 COMMERCIAL MINUS SIGN, U+2796 HEAVY MINUS SIGN (an emoji), U+2E43 DASH WITH LEFT UPTURN and U+10D8F GARAY MINUS SIGN.

Why

Tools like unidecode and anyascii transliterate every character, so café turns into cafe. That is the wrong tool when you only want to tidy punctuation and keep the words. smartypants goes the opposite way, turning ASCII into typographic dashes. unidash folds punctuation to ASCII, knows the whole Unicode dash family rather than the three most common members, and leaves letters alone. It is zero dependency and typed.

Development

The Unicode data files are not part of the repository. python scripts/fetch_vectors.py downloads them into tests/vectors/unicode/ and checks each against the size and SHA-256 in MANIFEST.txt; without them the data tests skip. The installed-wheel test runs with UNIDASH_E2E=1.

License

MIT. See LICENSE.

Links

• Website: https://lianes8studio.com
• GitHub: https://github.com/JaviLianes8
• LinkedIn: https://www.linkedin.com/in/jlianes/