Skip to main content

Different approach of chardet, this library goal is to read human text from unknown encoding and transpose it to unicode the best we can.

Project description

Welcome to Charset for Human 👋

License: MIT

Library that help you read human* written text from unknown charset encoding.
Project motivated by chardet, I'm trying to resolve the issue by taking another approach.

This project offer you a alternative to Universal Charset Encoding Detector, also known as Chardet. Also as of July, August 2019 it's still experimental. Use it with caution.

Feature Chardet Charset Normalizer
Fast**

🐌
Universal
Reliable without distinguishable standards
Reliable with distinguishable standards
Free & Open

Reading Normalized TextCat Reading Text

Cats are going to enjoy newly decoded text

Chardet has weaknesses where Charset Normalizer has not and vice versa. You could combine the strength of both lib to reach near perfect detection. 💪

** : Fast when there is distinguishable standards. * : When written, should not be gibberish.

Your support

Please ⭐ this repository if this project helped you!

✨ Installation

Using PyPi

pip install charset_normalizer

🚀 Basic Usage

Just print out normalized text

from charset_normalizer import CharsetNormalizerMatches as CnM

matches = CnM.from_path('./my_subtitle.srt')

if len(matches) > 0:
    print(
        str(matches.best().first())
    )

Convert any text file to UTF-8

from charset_normalizer import CharsetNormalizerMatches as CnM
try:
    CnM.normalize('./my_subtitle.srt') # should write to disk my_subtitle-***.srt
except IOError as e:
    print('Sadly, we are unable to perform charset normalization.', str(e))

See wiki for advanced usages. Todo, not yet available.

😇 Why

When I started using Chardet, I noticed that this library was wrong most of the time when it's not about Unicode, Gb or Big5. That because some charset are easily identifiable because of there standards and Chardet does a really good job at identifying them.

I don't care about the originating charset encoding, that because two different table can produce two identical file. What I want is to get readable text, the best I can.

In a way, I'm brute forcing text decoding. How cool is that ? 😎

🍰 How

  • Discard all charset encoding table that could not fit the binary content.
  • Measure chaos, or the mess once opened with a corresponding charset encoding.
  • Extract matches with the lowest mess detected.
  • Finally, if there is too much match left, we measure coherence.

Wait a minute, what is chaos/mess and coherence according to YOU ?

Chaos : I opened hundred of text files, written by humans, with the wrong encoding table. Then I observed, then I established some ground rules about what is obvious when it's seems like a mess. I know that my interpretation of what is chaotic is very subjective, feel free to contribute in order to improve or rewrite it.

Coherence : For each language there is on earth (the best we can), we have computed letter appearance occurrences ranked. So I thought that those intel are worth something here. So I use those records against decoded text to check if I can detect intelligent design.

👤 Contributing

Contributions, issues and feature requests are very much welcome.
Feel free to check issues page if you want to contribute.

📝 License

Copyright © 2019 Ahmed TAHRI @Ousret.
This project is MIT licensed.

Letter appearances frequencies used in this project © 2012 Denny Vrandečić

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

charset_normalizer-0.1a0.tar.gz (19.1 kB view details)

Uploaded Source

File details

Details for the file charset_normalizer-0.1a0.tar.gz.

File metadata

  • Download URL: charset_normalizer-0.1a0.tar.gz
  • Upload date:
  • Size: 19.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/40.8.0 requests-toolbelt/0.9.1 tqdm/4.32.2 CPython/3.7.3

File hashes

Hashes for charset_normalizer-0.1a0.tar.gz
Algorithm Hash digest
SHA256 2e9474d6ea0730c9e6b691423823fcc0a012ab5281e4cf451d047ccca593e185
MD5 fda35165d4cab813112ec19383271541
BLAKE2b-256 7e8dfaa8cc13b03896e65dcd0f67d56ef70f4ee9c14301f5ac4540c8aaaaf1aa

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page