terselib 0.1.0

Open-source implementation of the z/OS TERSE (AMATERSE / PACK / SPACK) compression format

TerseLib

A Python, implementation of the z/OS TERSE compression format (as produced by AMATERSE and the older TRSMAIN), written from scratch.

Compress files on Linux/macOS/Windows and unpack them with AMATERSE on the mainframe — or terse a data set on z/OS and decompress it anywhere.

• Both compression methods: PACK (LZW) and SPACK (LZMW) are supported
• Host (12-byte header) and native (4-byte header) containers
• Fixed (F/FB) and variable (V/VB) record formats, including RDW framing
• Text (EBCDIC ⇄ ASCII) and binary rendering, with selectable code pages
• Smart decompression: auto-detects text vs binary content and picks an output file extension by content sniffing (libmagic)
• Validated against the IBM tersedecompress test corpus: byte-identical output in both directions for every PACK/SPACK, FB/VB, text/binary combination (only exceptions: two corpus files that AMATERSE itself produced broken)

Why no off-the-shelf LZW library? TERSE's PACK is a nonstandard LZW: fixed 12-bit codes, a dictionary that starts full (pre-seeded with EBCDIC blanks) and least-recently-used slot recycling instead of a dictionary reset. SPACK is LZMW, which has no maintained Python implementation. Both codecs are therefore implemented here, straight from the spec. The ebcdic package (extra code pages) and python-magic (file-type detection) are used when installed.

Install

$ pip install terselib                # from PyPI; add [full] for ebcdic + magic
$ pip install terselib[full]

Or from a checkout (this directory contains pyproject.toml):

$ pip install .

Command line

$ terse report.txt                     # -> report.txt.trs (SPACK, text, VB)
$ terse -b image.png                   # binary, lossless (FB LRECL 1)
$ terse -m pack --recfm f -l 80 x.txt  # PACK, fixed 80-byte records

$ unterse DATA.TRS                     # auto: text/binary + extension sniffing
$ unterse -b DATA.TRS out.bin          # force exact binary output
$ unterse -t -c cp037 DATA.TRS out.txt # text via a specific code page
$ unterse -i DATA.TRS                  # show header info only

terselib compress|decompress|info is an umbrella alias for the same tools, also reachable as python -m terselib.

Moving files to/from z/OS

Transfer .trs files in binary (ftp> binary). On z/OS, allocate the receiving data set as RECFM=FB LRECL=1024 (the file size is always a multiple of 1024), then unpack with:

//UNPACK  EXEC PGM=AMATERSE,PARM='UNPACK'
//SYSUT1   DD DISP=SHR,DSN=YOUR.TERSED.FILE
//SYSUT2   DD DSN=YOUR.OUTPUT,DISP=(NEW,CATLG),...

Library

import terselib

# compress / decompress bytes
blob = terselib.compress(b"HELLO\nWORLD\n", text=True)     # SPACK, VB
result = terselib.decompress(blob)                          # auto-detect
assert result.data == b"HELLO\nWORLD\n" and result.text

# work with files; binary output gets a sniffed extension
terselib.compress_file("payload.bin", text=False, variable=False,
                        record_length=1)
path, result = terselib.decompress_file("payload.bin.trs")

# inspect a header without decompressing
print(terselib.read_header("DATA.TRS").describe())

# verbose logging (off by default)
terselib.enable_verbose_logging()

See the documentation (Sphinx / Read the Docs) for the full API reference.

Folder layout

.
├── pyproject.toml     packaging metadata; run pip install from here
├── src/terselib/      the library + terse/unterse/terselib CLIs
├── tests/              stdlib unittest suite
├── scripts/            corpus_check.py -- validates against a reference
│                        z/OS test corpus (see below)
└── docs/               Sphinx sources (readthedocs.io)

Development

$ python3 -m unittest discover -s tests            # unit tests
$ python3 scripts/corpus_check.py PATH/TO/tersedecompress-testdata \
      --skip-large --roundtrip                     # corpus validation

scripts/corpus_check.py needs a checkout of the tersedecompress project's tersedecompress-testdata (not included in this repo) — real AMATERSE-compressed files plus their z/OS-decompressed references, used to validate this implementation byte-for-byte in both directions.

License

GPL-3.0-or-later, see LICENSE.