Skip to main content

A Python port of the Apache Lucene ASCII Folding Filter that converts alphabetic, numeric, and symbolic Unicode characters which are not in the first 127 ASCII characters (the ‘Basic Latin’ Unicode block) into ASCII equivalents, if they exist.

Project description

fold_to_ascii

A Python port of the Apache Lucene ASCII Folding Filter that converts alphabetic, numeric, and symbolic Unicode characters which are not in the first 127 ASCII characters (the "Basic Latin" Unicode block) into ASCII equivalents, if they exist.

Usage

> from fold_to_ascii import fold
> s = u'Astroturf® paté'
> fold(s)
u'Astroturf pate'
> fold(s, u'?')
u'Astroturf? pate'

Differences from JS fold-to-ascii

This library always removes astral characters, even when a replacement character is specified. PRs welcome if you want to fix this.

Differences from unidecode

There are a few Python unidecode libraries out there (1, 2). They are based on a Perl program that makes some interesting choices about what to replace, for example, £ (POUND SIGN) is replaced with the string PS. Also, they do not allow specifying a replacement character to use other than the empty string for unmapped characters.

Development environment

$ virtualenv -p python3 ~/.local/venvs/fold_to_ascii
$ source ~/.local/venvs/fold_to_ascii/bin/activate
$ pip install -r requirements.txt

Test

$ make lint && make test

Release

Requires a pypi account.

  1. Bump the version in setup.py.
  2. Build and upload:
$ python setup.py sdist bdist_wheel
$ twine upload dist/*

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

fold_to_ascii-1.0.2.post1.tar.gz (22.7 kB view details)

Uploaded Source

Built Distribution

fold_to_ascii-1.0.2.post1-py3-none-any.whl (22.6 kB view details)

Uploaded Python 3

File details

Details for the file fold_to_ascii-1.0.2.post1.tar.gz.

File metadata

  • Download URL: fold_to_ascii-1.0.2.post1.tar.gz
  • Upload date:
  • Size: 22.7 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.46.0 CPython/3.7.7

File hashes

Hashes for fold_to_ascii-1.0.2.post1.tar.gz
Algorithm Hash digest
SHA256 7167a07fdc230bd5df5381e9221fe246d0eff3a86eb67e3935f91f5aeca3533a
MD5 0250fbcdb99a0448ae1aab769bf22c3a
BLAKE2b-256 71c43d131637f5b728b01c3ebcc982709384b43e6ccc521dd419d89f95308e51

See more details on using hashes here.

File details

Details for the file fold_to_ascii-1.0.2.post1-py3-none-any.whl.

File metadata

  • Download URL: fold_to_ascii-1.0.2.post1-py3-none-any.whl
  • Upload date:
  • Size: 22.6 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/3.1.1 pkginfo/1.5.0.1 requests/2.23.0 setuptools/46.1.3 requests-toolbelt/0.9.1 tqdm/4.46.0 CPython/3.7.7

File hashes

Hashes for fold_to_ascii-1.0.2.post1-py3-none-any.whl
Algorithm Hash digest
SHA256 6a06640b0cd8cdd16b83ba72a098fb4b243987a50b2f8213b419b6de97787de6
MD5 a55a7b7e25e027cf0fb5897035e5d124
BLAKE2b-256 045709fd562dd7ca24cf764c58ef53b590fcd15f6ec982ed3282fe588e55e64f

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page