Skip to main content

utf8cleaner

Read a file byte-by-byte to and strip any byte sequences that have no UTF8 equivalent and strip them

Installation

pip install utf8cleaner

Note: requires python 3

Usage

utf8cleaner --input FILENAME

Will read FILENAME and write to FILENAME.clean

Why would I want to do this?

Sometimes when exporting and importing data, there are byte sequences that prevent data being imported. To fix this you would otherwise have to do one or more of:

  • Manually edit the source data in its native application (eg backspace invisible characters) in JIRA fields
  • Edit the file with a hex editor and look for known-bad values (eg copyright symbol)
  • Do something smart with perl/vi/sed to find and replace known bad byte patterns

This simple utility fixes these problems in one hit.

Where do these strange characters come from?

Number one culprit: copy and paste from outlook. This often introduces invisible whitespace errors (spaces that are not spaces...) along with "pretty" quotes, etc.

Other sources including copying and pasting from files with the old ISO8859 character encodings

Can I see an example file that demonstrates this issue?

examples/test.txt

There is a copyright symbol at the end of the file that needs replacing

What exactly is the problem?

iso8859 represents symbols as a single byte, eg the copyright symbol would be represented by the single hex byte:

0xA9

UTF8 uses two bytes to represent such characters, eg:

0xC2 0xA9

Since UTF-8 is a variable width character encoding scheme, it will use from 1 to 4 bytes to encode a single symbol. This is how it is able to represent all kinds of new symbols we take for granted such as emojii and CJK characters.

Metadata

Release files for utf8cleaner 0.0.2

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for utf8cleaner 0.0.2
File Size Uploaded Provenance Actions
utf8cleaner-0.0.2.tar.gz 4.1 kB Without provenance Details for utf8cleaner-0.0.2.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for utf8cleaner 0.0.2
File Interpreter ABI Platform Actions
utf8cleaner-0.0.2-py3-none-any.whl Python 3 none any Details for utf8cleaner-0.0.2-py3-none-any.whl

Total release size: 8.8 kB

Release files / utf8cleaner-0.0.2.tar.gz

Download URL utf8cleaner-0.0.2.tar.gz
Size 4.1 kB
Tags Source
SHA-256 checksum
How to use checksums
eef956329bbf94352120b4893cad9b46f3fd32e25edd7be6c490df447531c7f0
BLAKE2b-256 checksum
How to use checksums
2ce8b5a11589825b5fd61475b45c055b68c1909a4829b6f2449e4496874e387b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/3.0.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/40.6.2 requests-toolbelt/0.9.1 tqdm/4.38.0 CPython/3.6.9

Release files / utf8cleaner-0.0.2-py3-none-any.whl

Download URL utf8cleaner-0.0.2-py3-none-any.whl
Size 4.7 kB
Tags Python 3
SHA-256 checksum
How to use checksums
bc447b12ebdedad15c1f18141e1375db417e0ed98b09016051eec731f36e7da6
BLAKE2b-256 checksum
How to use checksums
fa93a29d6f9aa750975527b11723ce13b2d64996e5f74c4436b415e54db6e4b6
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
No
Uploaded via twine/3.0.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/40.6.2 requests-toolbelt/0.9.1 tqdm/4.38.0 CPython/3.6.9

Security for utf8cleaner 0.0.2

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page