zlib-into 0.4

Compress & decompress data with zlib into preallocated buffers

zlib_into allows Python code to compress & decompress data into a preallocated buffer.

The zlib module in the Python standard library provides an interface to zlib, a common library for Deflate compression, but always automatically allocates memory for the output.

zlib_into defines two main functions:

compress_into(data, output, level=-1, wbits=15)

decompress_into(data, output, wbits=15)

In each case, output can be a bytearray, memoryview, Numpy array, or anything else compatible with Python’s buffer protocol exposing a contiguous, writable chunk of memory.

The other parameters have the same meanings as in zlib.compress and zlib.decompress.

This can be useful for:

• Decompressing regular chunks of a known size: you can allocate a single output array and fill each chunk directly, avoiding an extra copy.

• Compressing chunks when you want to skip compression if it would make the data larger: use a fixed-size buffer matching the input chunk size. Again you can avoid unnecessary memory allocations by reusing the buffer.

• Imposing a size limit on decompressed data: it will stop decompressing (and throw an error) when the buffer is full, instead of decompressing the whole thing to see how big it is. But note that security on untrusted input is not the primary goal of this library.

There are also two additional functions:

shuffle(data, output, itemsize)

unshuffle(data, output, itemsize)

Shuffle rearranges bytes in a way that often allows for better compression of numeric data: shuffling b"ababab" with an itemsize of 2 gives b"aaabbb". Unshuffle does the reverse. This is the same as the shuffle filter built in to HDF5.

For now, itemsize can only be 2, 4, or 8. The same operations can be done for any itemsize using NumPy views to transpose bytes, but the more specific code can be faster.