charex 0.2.3

A unicode and character set explorer.

charex is a Unicode and character set explorer for understanding issues with character set translation and Unicode normalization.

Why Did I Make This?

I find the ambiguity of text data interesting. In memory it’s all ones and zeros. There is nothing inherent to the data that makes 0x20 mean a space character, but we’ve mostly agreed that it does. That “mostly” part is what’s interesting to me, and it’s where a lot of fun problems lie.

How Do I Use This?

It’s in PyPI, so you can install it with pip, as long as you are using Python 3.11 or higher:

pip install charex

charex has four modes of operation:

• Direct command line invocation,

• An interactive shell,

• A graphical user interface (GUI),

• An application programming interface (API).

Command Line

To get help for direct invocation from the command line:

$ charex -h

Interactive Shell

To launch the interactive shell:

$ charex

That will bring you to the charex shell:

Welcome to the charex shell.
Press ? for a list of comands.

charex>

From here you can type ? to see the list of available commands:

Welcome to the charex shell.
Press ? for a list of comands.

charex> ?
The following commands are available:

  * cd: Decode the given address in all codecs.
  * ce: Encode the given character in all codecs.
  * cl: List registered character sets.
  * clear: Clear the terminal.
  * ct: Count denormalization results.
  * dm: Build a denormalization map.
  * dn: Perform denormalizations.
  * dt: Display details for a code point.
  * el: List the registered escape schemes.
  * es: Escape a string using the given scheme.
  * fl: List registered normalization forms.
  * nl: Perform normalizations.
  * sh: Run in an interactive shell.
  * up: List the Unicode properties.
  * uv: List the valid values for a Unicode property.

For help on individual commands, use "help {command}".

charex>

And then type help then a name of one of the commands to learn what it does:

charex> help dn
usage: charex dn [-h] [-m MAXDEPTH] [-n NUMBER] [-r] [-s SEED] form base

Denormalize a string.

positional arguments:
  form                  The normalization form for the denormalization. Valid
                        options are: casefold, nfc, nfd, nfkc, nfkd.
  base                  The base normalized string.

options:
  -h, --help            show this help message and exit
  -m MAXDEPTH, --maxdepth MAXDEPTH
                        Maximum number of reverse normalizations to use for
                        each character.
  -n NUMBER, --number NUMBER
                        Maximum number of results to return.
  -r, --random          Randomize the denormalization.
  -s SEED, --seed SEED  Seed the randomized denormalization.

charex>

GUI

To launch the charex GUI:

$ charex gui

API

To import charex into your Python script to get a summary of a Unicode character:

>>> import charex
>>>
>>>
>>> value = 'a'
>>> char = charex.Character(value)
>>> print(char.summarize())
a U+0061 (LATIN SMALL LETTER A)

What Version of Unicode Does This Support?

Parts of charex rely on unicodedata in the Python Standard Library. This limits charex to supporting the version supported by the version of Python you are running. There may be a bit of a lag as new Python versions are released, but as of this release of charex it supports:

• Python 3.11: Unicode 14.0

• Python 3.12: Unicode 15.0

• Python 3.13: Unicode 15.1

Support for Unicode 16.0 should come around the release of Python 3.14.

What Is Left To Do?

The following features are planned for the v0.2.4 or later releases:

• Support for Unicode v16.0 for Python 3.14.

• Emoji combiner.

• Basic doctests for all public classes and functions.

• Web API.

• Registration for character set codecs.

The list of Unicode properties can be found here: Index

The list of Unihan properties is here: tr38

Changes in v0.2.3

The following are the changes in v0.2.3:

• Move dependency management to poetry.

• Move package into /src directory.

• Use tox for regression testing.

• Support Unicode 15.0 for running under Python 3.12.

  • Add Unicode 15.0 files.

  • Added “kalternatetotalstrokes” property.

  • Moved “kcihait” property source.

  • Added “idna2008” property.

• Support Unicode 15.1 for running under Python 3.13.

  • Add Unicode 15.1 files.

  • Remove seven Unihan properties.

    • kHKSCS

    • kIRGDaiKanwaZiten

    • kKPS0

    • kKPS1

    • kKSC0

    • kKSC1

    • kRSKangXi

  • Add six Unihan properties.

    • kJapanese

    • kMojiJoho

    • kSMSZD2003Index

    • kSMSZD2003Readings

    • kVietnameseNumeric

    • kZhuangNumeric

  • Add “IDS_Unary_Operator” property.

  • Add “ID_Compat_Math_Start” property.

  • Add “ID_Compat_Math_Continue” property.

  • Add “NFKC_Simple_Casefold” property.

  • Check if multiple “kPrimaryNumeric” values are supported (see U+5146 and U+79ED).

• Use Unicode version supported by Python version.

  • Updated the path_map to handle different Unicode versions.

  • Updated the prop_map to handle different Unicode versions.

  • Can specify the Unicode version of a FileCache.

  • Create a FileCache for the Unicode version supported by the running Python on launch.

• Generate denormalizations for each version.

• Return an AttributeError rather than KeyError when an attribute doesn’t exist.

How Do I Run the Tests?

charex is using the pytest package for unit testing. It also comes with a makefile that automates testing. So, to run the tests:

• Install poetry: pip install poetry

• Install the development dependencies: poetry install

• To run just the unit tests: make test

• To run the full test suite: make pre

Common Problems

ModuleNotFoundError: No module name ‘_tkinter’ error

If you get the above error when running charex or its tests, it’s likely your Python install doesn’t have tkinter linked. How you fix it depends upon your Python install. If you are using Python 3.13 installed with homebrew on macOS, you can probably fix it with:

brew install python-tk@3.13