Skip to main content
Python Software Foundation 20th Year Anniversary Fundraiser  Donate today!

remove non-UTF8 bytes from an input file and write a cleaned up version

Project description


Read a file byte-by-byte to and strip any byte sequences that have no UTF8 equivalent and strip them


pip install utf8cleaner

Note: requires python 3


utf8cleaner --input FILENAME

Will read FILENAME and write to FILENAME.clean

Why would I want to do this?

Sometimes when exporting and importing data, there are byte sequences that prevent data being imported. To fix this you would otherwise have to do one or more of:

  • Manually edit the source data in its native application (eg backspace invisible characters) in JIRA fields
  • Edit the file with a hex editor and look for known-bad values (eg copyright symbol)
  • Do something smart with perl/vi/sed to find and replace known bad byte patterns

This simple utility fixes these problems in one hit.

Where do these strange characters come from?

Number one culprit: copy and paste from outlook. This often introduces invisible whitespace errors (spaces that are not spaces...) along with "pretty" quotes, etc.

Other sources including copying and pasting from files with the old ISO8859 character encodings

Can I see an example file that demonstrates this issue?


There is a copyright symbol at the end of the file that needs replacing

What exactly is the problem?

iso8859 represents symbols as a single byte, eg the copyright symbol would be represented by the single hex byte:


UTF8 uses two bytes to represent such characters, eg:

0xC2 0xA9

Since UTF-8 is a variable width character encoding scheme, it will use from 1 to 4 bytes to encode a single symbol. This is how it is able to represent all kinds of new symbols we take for granted such as emojii and CJK characters.

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for utf8cleaner, version 0.0.2
Filename, size File type Python version Upload date Hashes
Filename, size utf8cleaner-0.0.2-py3-none-any.whl (4.7 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size utf8cleaner-0.0.2.tar.gz (4.1 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page