Skip to main content

LZ4Frame library for Python (via C bindings)

Project description


This is an LZ4-frame compression library for Python v3.2+ (and 2.7+), bound to Yann Collet’s LZ4 C implementation.

Installing / packaging

# To get from PyPI
pip3 install py-lz4framed_ph4

# To only build extension modules inline (e.g. in repository)
python3 build_ext -i

# To build & install globally
python3 install

# To install locally with pip
pip install --upgrade --find-links=. .



This fork has several improvements I needed for my other project.

  • Streamed decompression continuation (on reconnect)
  • Streamed decompression state clone - checkpointing
  • Streamed decompression marshalling - failure recovery, checkpointing

More on improvements

The scenario improvements address is downloading & decompressing large LZ4 data stream on the fly (hundreds of GBs). If the download stream is interrupted the original decompressor had no way to resume the decompression where it stopped.


The main motivation is to recover from aforementioned interruptions. Decompressor object now supports changing of the file-like object that is read from. If input socket stream went down we can re-connect and continue from the position it stopped. More in test test_decompressor_fp_continuation.

State cloning

If the processing logic is more complex you can use clone_decompression_context to clone decompressor context (the whole decompression state) and revert to this checkpoint if something breaks. More in test test_decompressor_fp_clone.

State marshalling

In order to recover also from program crashes you can marshal / serialize the decompressor context to the (byte) string which can be later unmarshalled / deserialized and continue from that point. Marshalled state can be stored e.g., to a file. More in test test_decompressor_fp_marshalling.

Random access archive

Situation: 800 GB LZ4 encrypted file. You want random access the file so it can be map/reduced or processed in parallel from different offsets.

Marshalled decompressor state takes only the required amount of memory. If the state dump is performed on the block boundaries (i.e., when the size hint from the previous call was provided by the input stream) the marhsalled size would be only 184 B, in the best case scenario, 66 kB in the worse case - when LZ4 file is using linked mode.

Anyway, when state marshalling returns this small state the application can build a meta file, the mapping: position in the input stream -> decompressor context. With this meta file a new decompressor can jump to the particular checkpoint.


Single-function operation:

import lz4framed

compressed = lz4framed.compress(b'binary data')

uncompressed = lz4framed.decompress(compressed)

To iteratively compress (to a file or e.g. BytesIO instance):

with open('myFile', 'wb') as f:
    # Context automatically finalises frame on completion, unless an exception occurs
    with Compressor(f) as c:
            while (...):
        except Lz4FramedNoDataError:

To decompress from a file-like object:

with open('myFile', 'rb') as f:
        for chunk in Decompressor(f):
    except Lz4FramedNoDataError:
        # Compress frame data incomplete - error case

See also lz4framed/ for example usage.


import lz4framed
print(lz4framed.__version__, lz4framed.LZ4_VERSION, lz4framed.LZ4F_VERSION)

Command-line utility

python3 -mlz4framed
USAGE: lz4framed (compress|decompress) (INFILE|-) [OUTFILE]

(De)compresses an lz4 frame. Input is read from INFILE unless set to '-', in
which case stdin is used. If OUTFILE is not specified, output goes to stdout.



This library has been checked using flake8 and pylint, using a modified configuration - see pylint.rc and flake8.cfg.


python3 -m unittest discover -v .


The only existing lz4-frame interoperable implementation I was aware of at the time of writing (lz4tools) had the following limitations:

  • Incomplete implementation in terms of e.g. reference & memory leaks on failure
  • Lack of unit tests
  • Not thread safe
  • Does not release GIL during low level (de)compression operations
  • Did not address the requirements for an external project

Project details

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for py-lz4framed_ph4, version 0.9.11
Filename, size File type Python version Upload date Hashes
Filename, size py-lz4framed_ph4-0.9.11.tar.gz (93.0 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page