Skip to main content

Library for data compression using Nintendo's Yaz0 algorithm

Project description

syaz0

Library for data compression using the Yaz0 algorithm

syaz0 is a native module for Python 3.6+ that provides fast data compression and decompression using Nintendo's Yaz0 algorithm.

Performance

Decompression performance is on par with existing Yaz0 decoders.

As of late December 2019, syaz0 is able to compress files much faster than existing Yaz0 encoders. Files that are representative of Breath of the Wild assets were compressed 20x to 30x faster than with existing public tools for an equivalent or better compression ratio, and 70-80x faster (with a slightly worse ratio) in extreme cases.

At the default compression level, file sizes are typically within 1% of Nintendo's.

For detailed benchmarks, see the results files in the test directory.

Usage

syaz0 can be installed with pip3 install syaz0. Binary builds are provided for Windows 64 bits (only). On all other platforms, building from source is required. Skip to the end of the README for more information.

syaz0.get_header(data)

Returns a syaz0.Header corresponding to the Yaz0 file header with the fields magic, uncompressed_size, data_alignment, reserved.

syaz0.decompress(data)

Decompresses Yaz0-compressed data from a bytes-like object data. Returns a bytes object containing the uncompressed data.

syaz0.decompress_unsafe(data)

Decompresses Yaz0-compressed data from a bytes (not bytes-like) object data. Returns a bytes object containing the uncompressed data.

Unlike syaz0.decompress, this function assumes that the input data is well-formed. In exchange for slightly improved performance, no sanity checks are performed. Warning: Do not use on untrusted data.

syaz0.compress(data, data_alignment=0, level=7)

Compresses a bytes-like object data. Returns a bytes-like object containing the Yaz0-compressed data.

data_alignment is a hint for decoders to allocate buffers with the required data alignment. Defaults to 0, which indicates that no particular alignment is required.

level is the compression level (6-9). 6 is fastest and 9 is slowest. Higher compression levels result in better compression. 7 is a good compromise between compression ratio and performance.

Project information

syaz0 was written with two goals in mind: to improve performance for Yaz0 compression — which is excruciatingly slow if one also desires decent compression ratios — and to let me learn a bit more about compression.

After doing more research on compression algorithms and finding out just how similar Yaz0 and LZ77-style compression algorithms are, I tried to implement some common tricks to help improve the extremely poor compression performance due to the sliding window search.

But the implementation was still extremely slow compared to gzip. And probably badly implemented.

It turns out that many more tricks were needed for fast compression.

After stumbling upon zlib-ng and seeing how well-optimised it is and all the intrinsics I decided it was best not to reinvent the wheel. Thus syaz0 uses a copy of zlib-ng for all the heavy lifting (match searching). The following modifications were made:

  • The window size was reduced to 4K (2^12) to match Yaz0.
  • The compress function and the stream structures were changed to take a callback that is invoked every time a distance/length pair or a literal is emitted. (I'm not proud, but it works.)
  • MAX_MATCH was not increased. zlib assumes it is equal to 258 in too many places and increasing it actually gives worse compression ratios.

Building from source

Building syaz0 from source requires:

  • CMake 3.10+
  • A compiler that supports C++17
  • Everything needed to build zlib-ng
  • pybind11 2.4+ (including CMake config files)
  • setuptools

When no binary build is available, pip will automatically build from source during the install process.

To build from source manually, run python3 setup.py bdist_wheel.

License

This software is licensed under the terms of the GNU General Public License, version 2 or later.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

syaz0-1.0.1.tar.gz (704.2 kB view details)

Uploaded Source

Built Distributions

syaz0-1.0.1-cp38-cp38-win_amd64.whl (129.1 kB view details)

Uploaded CPython 3.8 Windows x86-64

syaz0-1.0.1-cp38-cp38-manylinux2014_x86_64.whl (147.7 kB view details)

Uploaded CPython 3.8

syaz0-1.0.1-cp37-cp37m-win_amd64.whl (129.1 kB view details)

Uploaded CPython 3.7m Windows x86-64

syaz0-1.0.1-cp37-cp37m-manylinux2014_x86_64.whl (150.0 kB view details)

Uploaded CPython 3.7m

syaz0-1.0.1-cp36-cp36m-win_amd64.whl (129.0 kB view details)

Uploaded CPython 3.6m Windows x86-64

syaz0-1.0.1-cp36-cp36m-manylinux2014_x86_64.whl (150.0 kB view details)

Uploaded CPython 3.6m

File details

Details for the file syaz0-1.0.1.tar.gz.

File metadata

  • Download URL: syaz0-1.0.1.tar.gz
  • Upload date:
  • Size: 704.2 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.4.0 requests-toolbelt/0.9.1 tqdm/4.32.2 CPython/3.7.5

File hashes

Hashes for syaz0-1.0.1.tar.gz
Algorithm Hash digest
SHA256 eb18b806b818b3ef7355d3700db1362d3aa210bac5ab0b9617e6d8ef474c7220
MD5 a58e90433768c6dae43c5f455fd29ab3
BLAKE2b-256 e49f6fcaf32537b4eb277dc24ccccabd2aaac2962772c845cfac64c953b6eaf7

See more details on using hashes here.

File details

Details for the file syaz0-1.0.1-cp38-cp38-win_amd64.whl.

File metadata

  • Download URL: syaz0-1.0.1-cp38-cp38-win_amd64.whl
  • Upload date:
  • Size: 129.1 kB
  • Tags: CPython 3.8, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.4.0 requests-toolbelt/0.9.1 tqdm/4.32.2 CPython/3.7.5

File hashes

Hashes for syaz0-1.0.1-cp38-cp38-win_amd64.whl
Algorithm Hash digest
SHA256 929be0e6a28ee7697bacb67969269cabb72cc83691e894151237bd37742d1f0d
MD5 07c606eda67ec2603f777bda2bd19ed6
BLAKE2b-256 759772f02d4bacd3052457c57c90b6b8cf9c02ac5ba0cae0e93805ea304a2d44

See more details on using hashes here.

File details

Details for the file syaz0-1.0.1-cp38-cp38-manylinux2014_x86_64.whl.

File metadata

  • Download URL: syaz0-1.0.1-cp38-cp38-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 147.7 kB
  • Tags: CPython 3.8
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.4.0 requests-toolbelt/0.9.1 tqdm/4.32.2 CPython/3.7.5

File hashes

Hashes for syaz0-1.0.1-cp38-cp38-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 3671d243ddb27f7a40cd105d19790e4bc80c81e55b1f2f5b7c72d46199ea1baf
MD5 da5bdded06226b4ad7680df345977198
BLAKE2b-256 7cd0f7d066dc1d2e815451a0e0ebefd1d58adb5707210b99ba6aa7085a26523d

See more details on using hashes here.

File details

Details for the file syaz0-1.0.1-cp37-cp37m-win_amd64.whl.

File metadata

  • Download URL: syaz0-1.0.1-cp37-cp37m-win_amd64.whl
  • Upload date:
  • Size: 129.1 kB
  • Tags: CPython 3.7m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.4.0 requests-toolbelt/0.9.1 tqdm/4.32.2 CPython/3.7.5

File hashes

Hashes for syaz0-1.0.1-cp37-cp37m-win_amd64.whl
Algorithm Hash digest
SHA256 5405ff967dd191a5e6e91c651b7cca50b5e1481861456307bcd67e6fb5f840cb
MD5 d8c3684148d851fd2357bb87e9477762
BLAKE2b-256 b08baee035c5487ae6924573ed90350dbc0e02cfb2e7a3fa08c3607c97ce337c

See more details on using hashes here.

File details

Details for the file syaz0-1.0.1-cp37-cp37m-manylinux2014_x86_64.whl.

File metadata

  • Download URL: syaz0-1.0.1-cp37-cp37m-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 150.0 kB
  • Tags: CPython 3.7m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.4.0 requests-toolbelt/0.9.1 tqdm/4.32.2 CPython/3.7.5

File hashes

Hashes for syaz0-1.0.1-cp37-cp37m-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 ed303d6e4516ced66391303c505a19ea2a7a8cdb04c5a70902c8676ea48e2137
MD5 e46b1559884ab1f14c6610c402af3e90
BLAKE2b-256 3ef699c1cdc2ae49db8eabea374f8013552430950e676958222b3884d4494049

See more details on using hashes here.

File details

Details for the file syaz0-1.0.1-cp36-cp36m-win_amd64.whl.

File metadata

  • Download URL: syaz0-1.0.1-cp36-cp36m-win_amd64.whl
  • Upload date:
  • Size: 129.0 kB
  • Tags: CPython 3.6m, Windows x86-64
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.4.0 requests-toolbelt/0.9.1 tqdm/4.32.2 CPython/3.7.5

File hashes

Hashes for syaz0-1.0.1-cp36-cp36m-win_amd64.whl
Algorithm Hash digest
SHA256 c0398bd9c520167e82e93e5dbec9107b44d0ef115ebe89fb147cb13ba78ecf3d
MD5 1917b9b73b956ba4b85c169db2a268e1
BLAKE2b-256 a642cbd5b0a394edad841159d4b63de237bac1a8c6e7dd8550adcaac4b98c69c

See more details on using hashes here.

File details

Details for the file syaz0-1.0.1-cp36-cp36m-manylinux2014_x86_64.whl.

File metadata

  • Download URL: syaz0-1.0.1-cp36-cp36m-manylinux2014_x86_64.whl
  • Upload date:
  • Size: 150.0 kB
  • Tags: CPython 3.6m
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.22.0 setuptools/41.4.0 requests-toolbelt/0.9.1 tqdm/4.32.2 CPython/3.7.5

File hashes

Hashes for syaz0-1.0.1-cp36-cp36m-manylinux2014_x86_64.whl
Algorithm Hash digest
SHA256 a2380dcbebe6e36bdbfe9a0dd9cf6efeb6ef15578b20a5ea789a939cc428b003
MD5 330a62e625b877eb2c174b7f7d3ad06d
BLAKE2b-256 f9eade8814245c2dbc0001a67e49e167f1bf9ba10163404c2d495165b8f824e8

See more details on using hashes here.

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page