Skip to main content

A Python port of the Apache Lucene ASCII Folding Filter that converts alphabetic, numeric, and symbolic Unicode characters which are not in the first 127 ASCII characters (the ‘Basic Latin’ Unicode block) into ASCII equivalents, if they exist.

Project description

fold_to_ascii

A Python port of the Apache Lucene ASCII Folding Filter that converts alphabetic, numeric, and symbolic Unicode characters which are not in the first 127 ASCII characters (the "Basic Latin" Unicode block) into ASCII equivalents, if they exist.

Usage

> from fold_to_ascii import fold
> s = u'Astroturf® paté'
> fold(s)
u'Astroturf pate'
> fold(s, u'?')
u'Astroturf? pate'

Differences from JS fold-to-ascii

This library always removes astral characters, even when a replacement character is specified. PRs welcome if you want to fix this.

Differences from unidecode

There are a few Python unidecode libraries out there (1, 2). They are based on a Perl program that makes some interesting choices about what to replace, for example, £ (POUND SIGN) is replaced with the string PS. Also, they do not allow specifying a replacement character to use other than the empty string for unmapped characters.

Development environment

$ virtualenv -p python3 ~/.local/venvs/fold_to_ascii
$ source ~/.local/venvs/fold_to_ascii/bin/activate
$ pip install -r requirements.txt

Test

$ make lint && make test

Release

Requires a pypi account.

  1. Bump the version in setup.py.
  2. Build and upload:
$ python setup.py sdist bdist_wheel
$ twine upload dist/*

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for fold-to-ascii, version 1.0.2.post1
Filename, size File type Python version Upload date Hashes
Filename, size fold_to_ascii-1.0.2.post1-py3-none-any.whl (22.6 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size fold_to_ascii-1.0.2.post1.tar.gz (22.7 kB) File type Source Python version None Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page