Standard Compression Scheme for Unicode

This package implements SCSU as a Python text codec.

Benefits of Unicode compression

Compressed strings typically have fewer bytes than strings encoded as UTF-8 or UTF-16.

Requirements

This package requires Python 3.10 or above.

Usage

Source code

Simply import the module and the SCSU codec is ready to use:

import scsu

b = s.encode("SCSU")

To automatically add and remove a byte-order mark signature, use SCSU-SIG instead of SCSU.

Command line interface

To compress a file, use the "encode" command: python3 -m scsu encode unicode.txt > scsu.txt

To decompress a file, use the "decode" command: python3 -m scsu decode scsu.txt > unicode.txt

To automatically add and remove a byte-order mark signature, add the -s option after the encode/decode command.

Errata

CPython bug #79792 causes the sample code (below) to not flush the encoding buffer:

with open(file, mode="w", encoding="SCSU-SIG") as f:
    f.write(s)  # Never flushes the encoding buffer.

A workaround is to import the codecs module, then replace open with codecs.open:

import codecs

with codecs.open(file, "w", encoding="SCSU-SIG") as f:
    f.write(s)  # Always flushes the encoding buffer.

Credits

Encoding logic is heavily based on a sample encoder described in "A survey of Unicode compression" by Doug Ewell and originally written by Richard Gillam in his book Unicode Demystified.

Enhancements to the encoding logic include:

• A two-character lookahead buffer to avoid a case where switching from Unicode to single-byte mode requires two window switches.
• Compression of sequential static window characters into a single new dynamic window, to avoid a case where a long string of punctuation is encoded as multiple quoted characters.
• Uses the Latin-1 Supplement window whenever possible so transcoding text encoded as ISO-8859-1 results in a valid SCSU and ISO-8859-1 byte string.

Decoding logic, however, is entirely original.