pigz 2.9rc2

pigz - python bindings to pigz (parallel gzip)

pigz

2.8 (19 Aug 2022) by Mark Adler

pigz, which stands for Parallel Implementation of GZip, is a fully functional replacement for gzip that exploits multiple processors and multiple cores to the hilt when compressing data.

pigz was written by Mark Adler and does not include third-party code. I am making my contributions to and distributions of this project solely in my personal capacity, and am not conveying any rights to any intellectual property of any third parties.

This version of pigz is written to be portable across Unix-style operating systems that provide the zlib and pthread libraries.

Type "make" in this directory to build the "pigz" executable. You can then install the executable wherever you like in your path (e.g. /usr/local/bin/). Type "pigz" to see the command help and all of the command options.

The latest version of pigz can be found at http://zlib.net/pigz/ . You need zlib version 1.2.3 or later to compile pigz. zlib version 1.2.6 or later is recommended, which reduces the overhead between blocks. You can find the latest version of zlib at http://zlib.net/ . You can look in pigz.c for the change history.

Questions, comments, bug reports, fixes, etc. can be emailed to Mark at his address in the license below.

The license from pigz.c is copied here:

This software is provided 'as-is', without any express or implied warranty. In no event will the author be held liable for any damages arising from the use of this software.

Permission is granted to anyone to use this software for any purpose, including commercial applications, and to alter it and redistribute it freely, subject to the following restrictions:

1. The origin of this software must not be misrepresented; you must not claim that you wrote the original software. If you use this software in a product, an acknowledgment in the product documentation would be appreciated but is not required.
2. Altered source versions must be plainly marked as such, and must not be misrepresented as being the original software.
3. This notice may not be removed or altered from any source distribution.

Mark Adler [e-mail]

Python Bindings

Python Bindings are powered by Pybind11 and the source code is available at https://github.com/thammegowda/pigz.

Install pigz

# Option 1: from PyPI
pip install pigz

# Option 2: from source code
pip install git+https://github.com/thammegowda/pigz

Features

1. Drop-in replacement for built in gzip (Aiming to be. Currently WIP)
2. Handles context managers. Automatically opens and closes files when used correctly with with block

Limitations

1. Python APIs are currently tailored for text mode
2. Maybe unstable on some platforms. Tested on GNU/Linux for now.

Usage

import pigz

data = "Hello, World!\nThis is a test\nThis is the third line"
out_path = 'test_iterator.gz'

# How to write data
with pigz.open(str(out_path), 'wt') as f:
   f.write(data)

# How to read data
with pigz.open(str(out_path), 'rt') as f:
   data2 = ''
   for line in f:
      data2 += line
assert data == data2

See tests/ directory for some more examples

Run Tests for Python Module

cmake -B build
cmake --build build
python -m pytest -vs tests/

Build Wheels for Release

NOTE: this for developers only

Many Linux Wheels

Build produces *-linux_x86_64.whl wheels but they are not readily portable. We need to convert them into portable manylinux*.whl files, which requires building on older toolchain (glibc)

Docs: https://github.com/pypa/manylinux

docker run -it -v $(pwd):/work quay.io/pypa/manylinux2014_x86_64 bash
cd /work  #go to the dir where pigz code is mounted

# build wheel for all supported python versions
for v in 3.{8..13}; do p=$(dirname $(realpath $(which python$v))); echo "rm -rf build; PATH=$p:$PATH python$v -m build -w"; done | bash
# python$v-config bin is required but it was unavailable in the default PATH, hence $p is added to enable python$v-config

# wheels are stored here
ls -lh dist/

# convert wheels to manylinux wheels
auditwheel repair dist/*.whl -w dist/wheelhouse


# Upload
# exit docker env
twine upload -r pypi dist/wheelhouse/*.whl