whatenc 0.4.1

Simple text encoding type classifier

whatenc

Text encoding type classifier.

whatenc is a command-line tool that uses a gradient-boosted tree classifier to detect the encoding of a given string or file.

The model is trained on text samples from the English, Greek, Russian, Hebrew, and Arabic Wikipedia corpora, chosen to represent a diverse set of writing systems (Latin, Greek, Cyrillic, Hebrew, and Arabic scripts). Each line is encoded using multiple encoding schemes to generate labeled examples.

How It Works

whatenc applies a feature-based approach to characterize text, then feeds these features into a gradient-boosted decision tree model to classify the encoding.

Feature Extraction

Each input string is converted into a feature vector describing its statistical properties.

Features include:

Feature	Description
Length (n)	Number of characters in the input
Alphabetic / Digit Ratios	Ratio of letters and digits to total length
Padding Ratio (=)	Common in Base64/32 encodings
Compressibility	Ratio of compressed to raw byte length
Shannon Entropy	Measure of randomness in single-character frequency distribution
Bigram Entropy	Measure of randomness in two-character (bigram) frequency distribution
Non-ASCII Ratio	Fraction of characters outside the ASCII range
Word Density	ratio of string length to word count

Supported Encodings

whatenc currently recognizes the following formats and transformations:

Category	Encodings
Base encodings	base32, base64, base85, hex, url
Text ciphers	morse
Compression	gzip64
Hash digests	md5, sha1, sha224, sha256, sha384, sha512

Installation

You can install whatenc using pipx:

pipx install whatenc

Usage

whatenc hello
whatenc samples.txt

Examples

[+] input: ZW5jb2RlIHRvIGJhc2U2NCBmb3JtYXQ=
   [~] top guess   = base64
      [=] base64   = 0.875
      [=] base32   = 0.101
      [=] gzip64   = 0.019

[+] input: hi
   [~] top guess   = plain
      [=] plain    = 0.772
      [=] base64   = 0.081
      [=] base32   = 0.075

[+] input: bfa99df33b137bc8fb5f5407d7e58da8
   [~] top guess   = md5
      [=] md5      = 1.000
      [=] sha1     = 0.000
      [=] url      = 0.000