charex 0.2.4

A unicode and character set explorer.

charex is a Unicode and character set explorer for understanding issues with character set translation and Unicode normalization.

Why Did I Make This?

I find the ambiguity of text data interesting. In memory it’s all ones and zeros. There is nothing inherent to the data that makes 0x20 mean a space character, but we’ve mostly agreed that it does. That “mostly” part is what’s interesting to me, and it’s where a lot of fun problems lie.

How Do I Use This?

It’s in PyPI, so you can install it with pip, as long as you are using Python 3.12 or higher:

pip install charex

charex has four modes of operation:

• Direct command line invocation,

• An interactive shell,

• A graphical user interface (GUI),

• An application programming interface (API).

Command Line

To get help for direct invocation from the command line:

$ charex -h

Interactive Shell

To launch the interactive shell:

$ charex

That will bring you to the charex shell:

Welcome to the charex shell.
Press ? for a list of comands.

charex>

From here you can type ? to see the list of available commands:

Welcome to the charex shell.
Press ? for a list of comands.

charex> ?
The following commands are available:

*  bl: Show the list of blocks.
*  cd: Decode the given address in all codecs.
*  ce: Encode the given character in all codecs.
*  cl: List registered character sets.
*  clear: Clear the terminal.
*  ct: Count denormalization results.
*  dm: Build a denormalization map.
*  dn: Perform denormalizations.
*  dt: Display details for a code point.
*  el: List the registered escape schemes.
*  es: Escape a string using the given scheme.
*  fl: List registered normalization forms.
*  gui: Start the GUI.
*  help: Display command list.
*  mf: Create emoji sequence for a region's flag.
*  nl: Perform normalizations.
*  ns: Show the list of named sequences.
*  pf: List characters with a given property value.
*  sh: Run in an interactive shell.
*  sv: Show the list of standardized variants.
*  up: List the Unicode properties.
*  uv: List the valid values for a Unicode property.
*  xt: Exit the charex shell.
*  zc: Show the list of emoji ZWJ sequence categories.
*  zl: Show the list of emoji ZWJ sequences.

For help on individual commands, use "help {command}".

charex>

And then type help then a name of one of the commands to learn what it does:

charex> help dn
usage: charex dn [-h] [-m MAXDEPTH] [-n NUMBER] [-r] [-s SEED] form base

Denormalize a string.

positional arguments:
  form                  The normalization form for the denormalization. Valid
                        options are: casefold, nfc, nfd, nfkc, nfkd.
  base                  The base normalized string.

options:
  -h, --help            show this help message and exit
  -m MAXDEPTH, --maxdepth MAXDEPTH
                        Maximum number of reverse normalizations to use for
                        each character.
  -n NUMBER, --number NUMBER
                        Maximum number of results to return.
  -r, --random          Randomize the denormalization.
  -s SEED, --seed SEED  Seed the randomized denormalization.

charex>

GUI

To launch the charex GUI:

$ charex gui

API

To import charex into your Python script to get a summary of a Unicode character:

>>> import charex
>>>
>>>
>>> value = 'a'
>>> char = charex.Character(value)
>>> print(char.summarize())
a U+0061 (LATIN SMALL LETTER A)

What Version of Unicode Does This Support?

Parts of charex rely on unicodedata in the Python Standard Library. This limits charex to supporting the version supported by the version of Python you are running. There may be a bit of a lag as new Python versions are released, but as of this release charex supports:

• Python 3.12: Unicode 15.0

• Python 3.13: Unicode 15.1

• Python 3.14: Unicode 16.0

What happened to Unicode 14.0?

I have the dependencies I use to generate the documentation in the development dependencies. To support Unicode 14.0, I had to support Python 3.11. To support Python 3.11, I had to use an old version of Sphinx that has some vulnerabilities. To clean that up, I had to drop support for Python 3.11 and Unicode 14.0.

Is there a way I could have fixed it without dropping Python 3.11 support? Probably. It’s is something I’ll try to look into when I get time. If it’s a problem for anyone, I’ll try to prioritize it.

What Is Left To Do?

The following features are planned for the v0.2.5 or later releases:

• Emoji combiner.

• Basic doctests for all public classes and functions.

• Web API.

• Registration for character set codecs.

• Allow unicode version switching.

The list of Unicode properties can be found here: Index

The list of Unihan properties is here: tr38

Changes in v0.2.4

The following are the changes in v0.2.4:

• Updated dependencies.

• Support for Unicode v16.0 for Python 3.14.

• Added support for the InCB property to Unicode v15.1.

• Removed official support for Python 3.11 to allow sphinx to be updated.

• Added some checking to see if Unicode properties had been mapped.

• Added alias_property() to the defined API.

• Added zalgo (glitch) text escape.

• Added the following commands:

  • bl: Shows the Unicode character blocks.

  • mf: Makes the flag for the given ISO 3166 region code,

  • zc: Lists the emoji ZWJ sequence categories.

Changes in v0.2.3

The following are the changes in v0.2.3:

• Move dependency management to poetry.

• Move package into /src directory.

• Use tox for regression testing.

• Support Unicode 15.0 for running under Python 3.12.

  • Add Unicode 15.0 files.

  • Added “kalternatetotalstrokes” property.

  • Moved “kcihait” property source.

  • Added “idna2008” property.

• Support Unicode 15.1 for running under Python 3.13.

  • Add Unicode 15.1 files.

  • Remove seven Unihan properties.

    • kHKSCS

    • kIRGDaiKanwaZiten

    • kKPS0

    • kKPS1

    • kKSC0

    • kKSC1

    • kRSKangXi

  • Add six Unihan properties.

    • kJapanese

    • kMojiJoho

    • kSMSZD2003Index

    • kSMSZD2003Readings

    • kVietnameseNumeric

    • kZhuangNumeric

  • Add “IDS_Unary_Operator” property.

  • Add “ID_Compat_Math_Start” property.

  • Add “ID_Compat_Math_Continue” property.

  • Add “NFKC_Simple_Casefold” property.

  • Check if multiple “kPrimaryNumeric” values are supported (see U+5146 and U+79ED).

• Use Unicode version supported by Python version.

  • Updated the path_map to handle different Unicode versions.

  • Updated the prop_map to handle different Unicode versions.

  • Can specify the Unicode version of a FileCache.

  • Create a FileCache for the Unicode version supported by the running Python on launch.

• Generate denormalizations for each version.

• Return an AttributeError rather than KeyError when an attribute doesn’t exist.

How Do I Run the Tests?

charex is using the pytest package for unit testing. It also comes with a makefile that automates testing. So, to run the tests:

• Install poetry: pip install poetry

• Install the development dependencies: poetry install

• To run just the unit tests: make test

• To run the full test suite: make pre

Common Problems

ModuleNotFoundError: No module name ‘_tkinter’ error

If you get the above error when running charex or its tests, it’s likely your Python install doesn’t have tkinter linked. How you fix it depends upon your Python install. If you are using Python 3.14 installed with homebrew on macOS, you can probably fix it with:

brew install python-tk@3.14