scsu 1.1

Standard Compression Scheme for Unicode

Standard Compression Scheme for Unicode

This package implements SCSU as a Python text codec.

Benefits of Unicode compression

Compressed strings typically have fewer bytes than strings encoded as UTF-8 or UTF-16.

Sample Text	In UTF-8	In UTF-16	In SCSU
¿Qué es Unicode?	18 bytes	32 bytes	16 bytes
Що таке Юнікод?	27 bytes	30 bytes	16 bytes
Ի՞նչ է Յունիկոդը ?	32 bytes	36 bytes	20 bytes
यूनिकोड क्या है?	42 bytes	32 bytes	17 bytes
ユニコードとは何か？	30 bytes	20 bytes	15 bytes
什麼是Unicode(統一碼/標準萬國碼)?	44 bytes	44 bytes	38 bytes
𐑢𐑳𐑑 𐑦𐑟 𐑿𐑯𐑦𐑒𐑴𐑛?	47 bytes	50 bytes	17 bytes
😀😁😂😃😄😅😆😇😈😉😊😋😌😍😎😏	64 bytes	64 bytes	19 bytes

Requirements

This package requires Python 3.10 or above.

Usage

Source code

Simply import the module and the SCSU codec is ready to use:

import scsu

b = s.encode("SCSU")

To automatically add and remove a byte-order mark signature, use SCSU-SIG instead of SCSU.

Command line interface

To compress a file, use the "encode" command: python3 -m scsu encode unicode.txt > scsu.txt

To decompress a file, use the "decode" command: python3 -m scsu decode scsu.txt > unicode.txt

To automatically add and remove a byte-order mark signature, add the -s option after the encode/decode command.

Errata

CPython bug #79792 causes the sample code (below) to not flush the encoding buffer:

with open(file, mode="w", encoding="SCSU-SIG") as f:
    f.write(s)  # Never flushes the encoding buffer.

A workaround is to import the codecs module, then replace open with codecs.open:

import codecs

with codecs.open(file, "w", encoding="SCSU-SIG") as f:
    f.write(s)  # Always flushes the encoding buffer.

Credits

Encoding logic is heavily based on a sample encoder described in "A survey of Unicode compression" by Doug Ewell and originally written by Richard Gillam in his book Unicode Demystified.

Enhancements to the encoding logic include:

• A two-character lookahead buffer to avoid a case where switching from Unicode to single-byte mode requires two window switches.
• Compression of sequential static window characters into a single new dynamic window, to avoid a case where a long string of punctuation is encoded as multiple quoted characters.
• Uses the Latin-1 Supplement window whenever possible so transcoding text encoded as ISO-8859-1 results in a valid SCSU and ISO-8859-1 byte string.

Decoding logic, however, is entirely original.