yez — maximum-ratio lossless compression

YEz is a from-scratch, multi-stage lossless compressor (C++17 core, Python API). It combines match parsing with a hash-chain/optimal parser, bitwise context mixing with 12 context models, x86/ARM64/delta filters, and a CRC32-verified container. No external compression library is used anywhere — pip install yez compiles the core from source on your machine.

Every archive is lossless and verified: the original data's CRC32 is stored and decompress() never returns anything that does not match byte-for-byte.

Developed by Ahmed Alhrrani — t.me/maho_s9.

Install

pip install yez

The package needs a C++17 compiler at install time; it picks the first available of c++, g++, clang++ (or $CXX if set). No prebuilt binaries are downloaded — the core is always compiled from the bundled sources:

# no system compiler? the ziglang pip package works too:
pip install ziglang
CXX="python -m ziglang c++" pip install yez

Quick start

import yez

z = yez.compress(b"hello " * 100000, level=9)   # -> bytes (a .yez archive)
data = yez.decompress(z)                         # -> exact original bytes

yez.compress_file("big.csv", "big.csv.yez", level=7)
yez.decompress_file("big.csv.yez", "big.csv")

info = yez.inspect(z)     # dict: version, mode, level, sizes, crc, blocks[]
ok = yez.check(z)         # structural validation without decompressing
yez.is_yez(z)             # magic-byte test

Measured results

Mixed 6.3 MB corpus of real files (text, CSV, JSON, XML, HTML), compared with Python's stdlib codecs on the same machine:

Per-file highlights (level 9 vs lzma -9e): text −25%, CSV −52%, JSON −28%, XML −38%, HTML −21%.

Levels

Level 9 is slow by design (context mixing is symmetric: decompression does the same modelling work, ≈0.1–0.2 MB/s each way). Level 5 decompresses at ~30 MB/s.

API reference

yez.compress(data, level=5, *, threads=None, block_size=0, block_crc=False) -> bytes
yez.decompress(data, *, threads=None) -> bytes
yez.compress_file(src, dst=None, level=5, *, threads=None, block_size=0, block_crc=False) -> int
yez.decompress_file(src, dst=None, *, threads=None) -> int
yez.inspect(data) -> dict
yez.check(data) -> bool
yez.is_yez(data) -> bool
yez.YezError                    # raised on corruption / bad input
yez.MAGIC                      # b"YEz"
yez.__version__

• Accepts bytes, bytearray and memoryview.
• threads=None picks automatically: all cores for levels 1–6, capped to 3 concurrent blocks for 7–9 (each context-mixing block holds ~400 MB of model state — the cap bounds peak memory; pass an explicit number to override).
• block_size (bytes; 0 = level default 4/16/32 MiB) and block_crc=True (per-block CRC32 — pinpoints which block is damaged) are available for tuning.
• Calls release the GIL, so one worker per thread scales from Python.
• yez.YezError is raised for bad magic, unsupported version, truncation, trailing data, header checksum mismatch or payload corruption — corrupt archives never yield wrong output silently.

Command line

yez c [-c N] [-t N] [-x] input output.yez    compress
yez d [-t N] input.yez output                decompress
yez t input.yez                              test integrity
yez i input.yez                              archive information

The same interface works as python -m yez ….

How it works

Input is split into blocks; each block is analysed (constancy, entropy, text-likeness, record stride, x86/ARM64 filter scores), candidate transforms are ranked with a cheap proxy parse, then the block is encoded with the best of: raw / constant / repstream (match parsing with a context-modelled binary range coder: greedy, lazy or price-optimal parsing, per-block lc/pb tuning) / ctxmix (bitwise context mixing: 12 hashed context models — orders 1–7, a UTF-8 aware word model, sparse, record-stride and an indirect model — two gated mixers and two APM/SSE stages). The smallest actual output wins; incompressible data never expands by more than ~25 bytes.

The binding is plain ctypes over a C ABI (yez_compress, yez_decompress, yez_check, yez_inspect_json, yez_free, yez_last_error), so the same core is usable from any FFI.

License

MIT — see LICENSE. Developed by Ahmed Alhrrani (t.me/maho_s9).