Skip to main content

Python Fast Unidecode

Build Python version Tests License: MIT


This repo is a fork of the rust-unicode repository and transports the original Rust implementation to be used with Python. It also implements a couple of source code changes to hasten a translation of ASCII family of characters and makes this implementation on par with Python unidecode implementation on this set of characters.

The overall result is this package should provide you with the same output as the aforementioned Python implementation. However, this package is much faster on a translation of non-ASCII characters (>~3x) and comparable to slightly slowe on ASCII characters (in a degree of small percents) on average based on the benchmark/speed_benchmark.py benchmark (depending on caching, etc.; sometimes, a translation of non-ASCII characters provides you with a speedup of up to >10x). The benchmarks were run on Python 3.13.

License

This project is licensed under the MIT License.

Important Note: Unlike the original Python unidecode package, which is distributed under the restrictive GNU General Public License (GPL), fast-unidecode is released under the permissive MIT license. This makes it suitable for use in a wider range of projects, including commercial and closed-source applications. For SaaS (Software as a Service) companies, using a GPL-licensed library can create an obligation to release your own source code, a requirement that the MIT license does not have.

Benchmark code is not a part of the distributed package.

Installation

pip install fast_unidecode
Installation from source

First, you need to build the package using maturin, then install fast_unidecode simply with pip.

maturin build --release
pip install target/wheels/fast_unidecode...

Usage

>>> from fast_unidecode import unidecode

>>> print(unidecode("Æneid"))
'AEneid'

>>> print(unidecode("北亰"))
'Bei Jing'
rust-unidecode (Original README.md)

Documentation

The rust-unidecode library is a Rust port of Sean M. Burke's famous Text::Unidecode module for Perl. It transliterates Unicode strings such as "Æneid" into pure ASCII ones such as "AEneid." For a detailed explanation on the rationale behind using such a library, you can refer to both the documentation of the original module and this article written by Burke in 2001.

The data set used to translate the Unicode was ported directly from the Text::Unidecode module using a Perl script, so rust-unidecode should produce identical output.

Examples

extern crate unidecode;
use unidecode::unidecode;

assert_eq!(unidecode("Æneid"), "AEneid");
assert_eq!(unidecode("étude"), "etude");
assert_eq!(unidecode("北亰"), "Bei Jing");
assert_eq!(unidecode("ᔕᓇᓇ"), "shanana");
assert_eq!(unidecode("げんまい茶"), "genmaiCha ");

Guarantees and Warnings

Here are some guarantees you have when calling unidecode():

  • The String returned will be valid ASCII; the decimal representation of every char in the string will be between 0 and 127, inclusive.
  • Every ASCII character (0x0000 - 0x007F) is mapped to itself.
  • All Unicode characters will translate to a string containing newlines ("\n") or ASCII characters in the range 0x0020 - 0x007E. So for example, no Unicode character will translate to \u{01}. The exception is if the ASCII character itself is passed in, in which case it will be mapped to itself. (So '\u{01}' will be mapped to "\u{01}".)

There are, however, some things you should keep in mind:

  • As stated, some transliterations do produce \n characters.
  • Some Unicode characters transliterate to an empty string, either on purpose or because rust-unidecode does not know about the character.
  • Some Unicode characters are unknown and transliterate to "[?]".
  • Many Unicode characters transliterate to multi-character strings. For example, 北 is transliterated as "Bei ".

This information was paraphrased from the original Text::Unidecode documentation.

Metadata

Release files for fast-unidecode 1.0.3.post1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Built distributions (wheels)

Table of built distributions (wheels) for fast-unidecode 1.0.3.post1
File
fast_unidecode-1.0.3.post1-cp314-cp314-manylinux_2_28_x86_64.whl CPython 3.14 CPython 3.14 Linux glibc 2.28+ x86-64 Details
fast_unidecode-1.0.3.post1-cp314-cp314-manylinux_2_28_aarch64.whl CPython 3.14 CPython 3.14 Linux glibc 2.28+ ARM64 Details
fast_unidecode-1.0.3.post1-cp314-cp314-macosx_11_0_arm64.whl CPython 3.14 CPython 3.14 macOS 11.0+ ARM64 Details
fast_unidecode-1.0.3.post1-cp313-cp313-manylinux_2_28_x86_64.whl CPython 3.13 CPython 3.13 Linux glibc 2.28+ x86-64 Details
fast_unidecode-1.0.3.post1-cp313-cp313-manylinux_2_28_aarch64.whl CPython 3.13 CPython 3.13 Linux glibc 2.28+ ARM64 Details
fast_unidecode-1.0.3.post1-cp313-cp313-macosx_11_0_arm64.whl CPython 3.13 CPython 3.13 macOS 11.0+ ARM64 Details
fast_unidecode-1.0.3.post1-cp312-cp312-manylinux_2_28_x86_64.whl CPython 3.12 CPython 3.12 Linux glibc 2.28+ x86-64 Details
fast_unidecode-1.0.3.post1-cp312-cp312-manylinux_2_28_aarch64.whl CPython 3.12 CPython 3.12 Linux glibc 2.28+ ARM64 Details
fast_unidecode-1.0.3.post1-cp312-cp312-macosx_11_0_arm64.whl CPython 3.12 CPython 3.12 macOS 11.0+ ARM64 Details
fast_unidecode-1.0.3.post1-cp311-cp311-manylinux_2_28_x86_64.whl CPython 3.11 CPython 3.11 Linux glibc 2.28+ x86-64 Details
fast_unidecode-1.0.3.post1-cp311-cp311-manylinux_2_28_aarch64.whl CPython 3.11 CPython 3.11 Linux glibc 2.28+ ARM64 Details
fast_unidecode-1.0.3.post1-cp311-cp311-macosx_11_0_arm64.whl CPython 3.11 CPython 3.11 macOS 11.0+ ARM64 Details

Total release size: 5.6 MB

Release files / fast_unidecode-1.0.3.post1-cp314-cp314-manylinux_2_28_x86_64.whl

Download URL fast_unidecode-1.0.3.post1-cp314-cp314-manylinux_2_28_x86_64.whl
Size 494.5 kB
Tags CPython 3.14 Linux glibc 2.28+ x86-64
SHA-256 checksum
How to use checksums
f9637691cfe605474053280827692714d52f7f037c27e8225f7080bbcafb2046
BLAKE2b-256 checksum
How to use checksums
363052b00198751b92259ba350e3e792abca927d05c41751c10325e47311770a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.13.12

Release files / fast_unidecode-1.0.3.post1-cp314-cp314-manylinux_2_28_aarch64.whl

Download URL fast_unidecode-1.0.3.post1-cp314-cp314-manylinux_2_28_aarch64.whl
Size 570.1 kB
Tags CPython 3.14 Linux glibc 2.28+ ARM64
SHA-256 checksum
How to use checksums
0006541e9b24318803e2c11c224c7698f3a2332864212524ede422192668d4ac
BLAKE2b-256 checksum
How to use checksums
283802a329a6b42abc559d3e2fe4a386e305f8190b7010e8897b472072296534
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.13.12

Release files / fast_unidecode-1.0.3.post1-cp314-cp314-macosx_11_0_arm64.whl

Download URL fast_unidecode-1.0.3.post1-cp314-cp314-macosx_11_0_arm64.whl
Size 324.0 kB
Tags CPython 3.14 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
15a3c70d4e55422aeaac5772c58f1aa1565c21ff40a278802e55828c617549b8
BLAKE2b-256 checksum
How to use checksums
dda66ca4175e29b7d3ce1903076b6e9ffe5bd67be845034558389fd3833af82d
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.13.12

Release files / fast_unidecode-1.0.3.post1-cp313-cp313-manylinux_2_28_x86_64.whl

Download URL fast_unidecode-1.0.3.post1-cp313-cp313-manylinux_2_28_x86_64.whl
Size 494.7 kB
Tags CPython 3.13 Linux glibc 2.28+ x86-64
SHA-256 checksum
How to use checksums
fd7ced585b10e562727e9a22c9e9c3574f9e17fc3636119e5e7a5b7c2af35221
BLAKE2b-256 checksum
How to use checksums
1d393928be0c805fbbb921953f55eb8ec04b5c7a80521a69966ed85ebf79555b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.13.12

Release files / fast_unidecode-1.0.3.post1-cp313-cp313-manylinux_2_28_aarch64.whl

Download URL fast_unidecode-1.0.3.post1-cp313-cp313-manylinux_2_28_aarch64.whl
Size 569.7 kB
Tags CPython 3.13 Linux glibc 2.28+ ARM64
SHA-256 checksum
How to use checksums
566829712084383157c61951282b70c1010d705d64aca7f986f8eaefc9c6cd68
BLAKE2b-256 checksum
How to use checksums
19f08407ee41d90ca5f63decbe0bd465e8511f0053d72d82260393bd349eb080
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.13.12

Release files / fast_unidecode-1.0.3.post1-cp313-cp313-macosx_11_0_arm64.whl

Download URL fast_unidecode-1.0.3.post1-cp313-cp313-macosx_11_0_arm64.whl
Size 324.0 kB
Tags CPython 3.13 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
c8174ec63be4fe8cf1f5f5fdeb08ec4777e9de42b83bf0513f0b4b27f45e102a
BLAKE2b-256 checksum
How to use checksums
53ae5a415e21b0b17483714b5db6c42cd18588553c81ada7ab51fb243109dddd
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.13.12

Release files / fast_unidecode-1.0.3.post1-cp312-cp312-manylinux_2_28_x86_64.whl

Download URL fast_unidecode-1.0.3.post1-cp312-cp312-manylinux_2_28_x86_64.whl
Size 495.1 kB
Tags CPython 3.12 Linux glibc 2.28+ x86-64
SHA-256 checksum
How to use checksums
370c81c03bd82cb6702643b0841413b51aa8ba7a1e8fa8994ee4644fbeb11bab
BLAKE2b-256 checksum
How to use checksums
cb0104cc234108e0ddaf5d22d3c47ec0f28ca254e522503db0786fb75f26f3c4
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.13.12

Release files / fast_unidecode-1.0.3.post1-cp312-cp312-manylinux_2_28_aarch64.whl

Download URL fast_unidecode-1.0.3.post1-cp312-cp312-manylinux_2_28_aarch64.whl
Size 569.9 kB
Tags CPython 3.12 Linux glibc 2.28+ ARM64
SHA-256 checksum
How to use checksums
186090d406c83c33bad18f7c25b05240bdac3c320793d87a4d2dcc468091bf63
BLAKE2b-256 checksum
How to use checksums
f29126f9be6465aad4231bf657ebd2e502b7a980e9f8f0653abf39e8de526a9c
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.13.12

Release files / fast_unidecode-1.0.3.post1-cp312-cp312-macosx_11_0_arm64.whl

Download URL fast_unidecode-1.0.3.post1-cp312-cp312-macosx_11_0_arm64.whl
Size 323.7 kB
Tags CPython 3.12 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
bcf112b8ec2e83012528c2d14658f06dd8ac6344efdef42eccb1b0bdc8bf3dee
BLAKE2b-256 checksum
How to use checksums
9539bc804bfdb4de77802825ba04e18c53184e8e40271ed945691cc928f8512b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.13.12

Release files / fast_unidecode-1.0.3.post1-cp311-cp311-manylinux_2_28_x86_64.whl

Download URL fast_unidecode-1.0.3.post1-cp311-cp311-manylinux_2_28_x86_64.whl
Size 495.3 kB
Tags CPython 3.11 Linux glibc 2.28+ x86-64
SHA-256 checksum
How to use checksums
17c1fd9dbed18894e9f8e491e79bbf69215e371177e07392a24f48fac3a52277
BLAKE2b-256 checksum
How to use checksums
6ab8d61761869a42dd2aef04686a0124f436afcf264cae983361c000dd9e76a4
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.13.12

Release files / fast_unidecode-1.0.3.post1-cp311-cp311-manylinux_2_28_aarch64.whl

Download URL fast_unidecode-1.0.3.post1-cp311-cp311-manylinux_2_28_aarch64.whl
Size 570.5 kB
Tags CPython 3.11 Linux glibc 2.28+ ARM64
SHA-256 checksum
How to use checksums
d28e46ab193b8bbffb607ca3d4b0c5a55dc63a6882ef55b16bb70e6ca102ee69
BLAKE2b-256 checksum
How to use checksums
ced539885ff8ad2536dfe7689b672fcff3f4d98d03f5aeed83eb93b6028195dc
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.13.12

Release files / fast_unidecode-1.0.3.post1-cp311-cp311-macosx_11_0_arm64.whl

Download URL fast_unidecode-1.0.3.post1-cp311-cp311-macosx_11_0_arm64.whl
Size 324.9 kB
Tags CPython 3.11 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
04fa367b20e9ca2e0a5d00a3dbaeee6131196f5eaf035fb35e7e6267facbe571
BLAKE2b-256 checksum
How to use checksums
e40226fdfa2bb8ae4699062752f5bf0c1b0ead6732047deb2d5ec6dfd1c93cb0
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/6.1.0 CPython/3.13.12

Release history Release notifications | RSS feed

This release

1.0.3.post1 This release

12 release files

1.0.0

8 release files

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page