Check if Unicode text files are Unicode-normalized
Project description
Unicodecheck
Simple tool to check if Unicode text files are Unicode-normalized
Install
pip3 install unicodecheck
Usage
Quickstart
unicodecheck -iv SPAM.txt
To check files in a directory recursively:
unicodecheck -ivr Ham/Eggs/
Synopsis
The main program can be invoked either through the unicodecheck
command or through the Python main module option python3 -m unicodecheck
.
usage: unicodecheck [-h] [-V] [-m {NFC,NFD,NFKC,NFKD}] [-d] [-u [NUMBER]] [-r] [-i] [-v]
PATH [PATH ...]
positional arguments:
PATH describe input file or directory (pass '-' to specify stdin)
options:
-h, --help show this help message and exit
-V, --version show program's version number and exit
-m {NFC,NFD,NFKC,NFKD}, --mode {NFC,NFD,NFKC,NFKD}
target Unicode normalization (default: NFC)
-d, --diff show diffs between the original and normalized (default: False)
-u [NUMBER], -U [NUMBER], --unified [NUMBER]
show unified diffs with NUMBER lines of context [NUMBER=3] (default: False)
-r, --recursive follow the directory tree rooted in each PATH argument (default: False)
-i, --include-hidden include hidden files and directories (default: False)
-v, --verbose report non-essential logs (default: False)
Tips
Check whether filenames are normalized
convmv
command is suitable.
NFC
convmv -f utf8 -t utf8 --nfc -r ./
NFD
convmv -f utf8 -t utf8 --nfd -r ./
Notes
- This tool doesn't provide auto in-place (write) file normalization because Unicode normalization doesn't guarantee content equivalence.
- The procedure for determining the binary file refers to Git's algorithm.
License
MIT
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
unicodecheck-1.0.2.tar.gz
(7.7 kB
view hashes)
Built Distribution
Close
Hashes for unicodecheck-1.0.2-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 015f1cbb031ccb775407817f92876d6f72047a8e293be1765583d09b087d2158 |
|
MD5 | 74cbdde47137a157bdc0f5b8a3099b7c |
|
BLAKE2b-256 | 331ea7cf172bf8f92eb9d843f381c0a357a4948f3092ade0d3d3c3bec75be625 |