zstd 1.5.7.3

ZSTD Bindings for Python

wheels for:	status
cpython 2.7 x86 u18
cpython 2.7 x86 u18 ext
cpython 2.7 x64 u18
cpython 2.7 armhf u18
cpython 3.4 x86 u16
cpython 3.4 x64 u16
cpython 3.5 x86 u16
cpython 3.5 x64 u16
cpython 3.6 x86 u16
cpython 3.6 x64 u16
cpython 3.7 x86 u16
cpython 3.7 x64 u16
cpython 3.8 x86 u16
cpython 3.8 x64 u16
cpython 3.9 x86 u16
cpython 3.9 x64 u16
cpython 3.10 x86 u16
cpython 3.10 x64 u16
cpython 3.11 x86 u16
cpython 3.11 x64 u16
cpython 3.12 x86 u16
cpython 3.12 x64 u16
cpython 3.13 x64 u22
cpython 3.13 x86 u22
cpython 3.13 noGIL x64 u22
cpython 3.13 noGIL x86 u16
cpython 3.14 x64 u16
cpython 3.14 noGIL x64 u16
cpython 3.14 noGIL x86 u16
cpython 3.14 x86 u16
cpython 3.15 x64 u16
cpython 3.15 x86 u16
cpython 3.15 noGIL x64 u16
cpython 3.15 noGIL x86 u16
Release 1.5.7.2
Master

Simple python bindings to Yann Collet ZSTD compression library.

Zstd, short for Zstandard, is a new lossless compression algorithm,

which provides both good compression ratio and speed for your standard compression needs. “Standard” translates into everyday situations which neither look for highest possible ratio (which LZMA and ZPAQ cover) nor extreme speeds (which LZ4 covers).

It is provided as a BSD-license package, hosted on GitHub.

WARNING!!!

If you setup 1.0.0.99.1 version - remove it manualy to able to update. PIP matching version strings not tuple of numbers.

Result generated by versions prior to 1.0.0.99.1 is not compatible with orignial Zstd by any means. It generates custom header and can be read only by zstd python module.

As of 1.0.0.99.1 version it uses standard Zstd output, not modified.

To prevent data loss there is two functions now: `compress_old` and `decompress_old`. They are works just like in old versions prior to 1.0.0.99.1.

As of 1.1.4 version module build without them by default.

As of 1.3.4 version these functions are deprecated and will be removed in future releases.

As of 1.5.0 version these functions are removed.

DISCLAIMER

These python bindings are kept simple and blunt.

Support of dictionaries and streaming is not planned.

LINKS

• Zstandard: https://github.com/facebook/zstd

• More full-featured and compatible with Zstandard python bindings by Gregory Szorc: https://github.com/indygreg/python-zstandard

Build from source

You need module setuptools <72 to run tests:

>>> $ pip install -U 'setuptools<72'

>>> $ git clone https://github.com/sergey-dryabzhinsky/python-zstd
>>> $ git submodule update --init
>>> $ apt-get install python-dev python3-dev python-setuptools python3-setuptools
>>> $ python setup.py build_ext clean
>>> $ python3 setup.py build_ext clean

And you need to install libzstd developer files at least version 1.4.0:

>>> $ dnf install -y libzstd-devel
# or
>>> $ apt install -y libzstd-dev
# or
>>> $ apk add zstd-dev

or do manual installation of zstd from source.

And you need C99 support in compiler (gcc >= 4.8), libc >= 2.14.

Note: Zstd legacy format support disabled by default. To build with Zstd legacy versions support - pass --legacy option to setup.py script:

>>> $ python setup.py build_ext --legacy clean

When using a PEP 517 builder you can use ZSTD_LEGACY environment variable instead:

>>> $ ZSTD_LEGACY=1 python -m build -w

Note: Python-Zstd legacy format support removed since 1.5.0. If you need to convert old data - checkout 1.4.9.1 module version. Support of it disabled by default. To build with python-zstd legacy format support (pre 1.1.2) - pass --pyzstd-legacy option to setup.py script:

>>> $ python setup.py build_ext --pyzstd-legacy clean

But beware! Legacy formats support state is unknown in this case. And if your version not equal with python-zstd - tests may not pass.

If you’re scared of threads you may pass option –libzstd-no-threads:

>>> $ python setup.py build_ext --libzstd-no-threads clean

When using a PEP 517 builder you can use ZSTD_THREADS environment variable instead:

>>> $ ZSTD_THREADS=0 python -m build -w

If you’re want to enable some speedup (maybe) you may try to enable built-in optimizations and pass option –libzstd-use-asm:

>>> $ python setup.py build_ext --libzstd-use-asm clean

Or add more speed with option –libzstd-use-asm-bmi2 to use instructions for new AMD CPU. When using a PEP 517 builder you can use ZSTD_ASM environment variable instead: And ZST_ASM_BMI2=1 too for bmi2 use.

>>> $ ZSTD_ASM=1 python -m build -w

If you want to build smaller module by size try to use option –small, but it will work slower.

>>> $ python setup.py build_ext --small clean

When using a PEP 517 builder you can use ZSTD_SMALL environment variable instead:

>>> $ ZSTD_SMALL=1 python -m build -w

If you want to build faster module try to use options –speed3, –speed1, –speed2, which corresponds with gcc options -O3, -O1, -O2.

>>> $ python setup.py build_ext --speed1 clean

When using a PEP 517 builder you can use ZSTD_SPEED3 (default), ZSTD_SPEED1, ZSTD_SPEED2 environment variables instead:

>>> $ ZSTD_SPEED2=1 python -m build -w

If you want to build even faster module try to use option –speed-max, but it will be optimized to your procesor only, similar to gcc options -O3 -march=native.

>>> $ python setup.py build_ext --speed-max clean

When using a PEP 517 builder you can use ZSTD_SPEEDMAX environment variable instead:

>>> $ ZSTD_SPEEDMAX=1 python -m build -w

If you want to build with existing distribution of libzstd just add --external option

>>> $ python setup.py build_ext --external clean

When using a PEP 517 builder you can use ZSTD_EXTERNAL environment variable instead:

>>> $ ZSTD_EXTERNAL=1 python -m build -w

If you want to build with a lot of debug output to stderr just add --debug option

>>> $ python setup.py build_ext --debug clean

When using a PEP 517 builder you can use ZSTD_DEBUG environment variable instead:

>>> $ ZSTD_DEBUG=1 python -m build -w

If you want to build with a lot more of debug output to stderr just add --debug-notice option

>>> $ python setup.py build_ext --debug-notice clean

When using a PEP 517 builder you can use ZSTD_DEBUG_NOTICE environment variable instead:

>>> $ ZSTD_DEBUG_NOTICE=1 python -m build -w

If you want to build with a lot more of debug output to stderr just add --debug-info option

>>> $ python setup.py build_ext --debug-info clean

When using a PEP 517 builder you can use ZSTD_DEBUG_INFO environment variable instead:

>>> $ ZSTD_DEBUG_INFO=1 python -m build -w

Some python builds need to force disabing LTO, so just add --force-no-lto option

>>> $ python setup.py build_ext --force-no-lto clean

When using a PEP 517 builder you can use ZSTD_BUILD_NO_LTO environment variable instead:

>>> $ ZSTD_BUILD_NO_LTO=1 python -m build -w

Some python builds need to force enabling stripping binary of the module, so just add --force-strip option

>>> $ python setup.py build_ext --force-strip clean

When using a PEP 517 builder you can use ZSTD_BUILD_STRIP environment variable instead:

>>> $ ZSTD_BUILD_STRIP=1 python -m build -w

If paths to header file zstd.h and libraries is uncommon - use common build params: –libraries –include-dirs –library-dirs.

>>> $ python setup.py build_ext --external --include-dirs /opt/zstd/usr/include --libraries zstd --library-dirs /opt/zstd/lib clean

But If you want to force build with bundled distribution of libzstd just add -- libzstd-bundled option

>>> $ python setup.py build_ext --libzstd-bundled clean

When using a PEP 517 builder you can use ZSTD_BUNDLED environment variable instead:

>>> $ ZSTD_BUNDLED=1 python -m build -w

If you want to check if build w/o any warnings just add -- all-warnings option

>>> $ python setup.py build_ext --all-warnings clean

When using a PEP 517 builder you can use ZSTD_WARNINGS environment variable instead:

>>> $ ZSTD_WARNINGS=1 python -m build -w

If you want to treat all warnings as errors just add --all-warnings-errors option

>>> $ python setup.py build_ext --all-warnings-errors clean

When using a PEP 517 builder you can use ZSTD_WERRORS environment variable instead:

>>> $ ZSTD_WERRORS=1 python -m build -w

When using a PEP 517 builder you can use ZSTD_FULLTIME_TESTS environment variable, to run tests with fulltime length:

>>> $ ZSTD_FULLTIME_TESTS=1 python setup.py test

Install from pypi

>>> # for Python 2.7+
>>> $ pip install zstd
>>> # or for Python 3.4+
>>> $ pip3 install zstd

API

Error

Standard python Exception for zstd module

ZSTD_compress (data[, level, threads, strict]): string|bytes

Function, compress input data block via mutliple threads, return compressed block, or raises Error.

Params:

• data: string|bytes - input data block, length limited by 2Gb by Python API

• level: int - compression level, ultra-fast levels from -100 (ultra) to -1 (fast) available since zstd-1.3.4, and from 1 (fast) to 22 (slowest), 0 or unset - means default (3). Default - 3.

• threads: int - how many threads to use, from 0 to 200, 0 or unset - auto-tune by cpu cores count. Default - 0. Since: 1.4.4.1

• strict: int - strict behaviour, raise zstd.Error if threads number or compression level is beyond limitations. Default - 0. Since: 1.5.6.3

Aliases:

• compress(…),

• dumps(…),

• encode(…) since: 1.5.6.2

Exception if: - level bigger than max level

Max number of threads: - 32bit system: 64 - 64bit system: 256 If provided bigger number - silently set maximber (since 1.5.4.1)

Since: 0.1

ZSTD_uncompress (data): string|bytes

Function, decompress input compressed data block, return decompressed block, or raises Error.

Support compressed data with multiple/concatenated frames (blocks) (since 1.5.5.1).

Support streamed data, since 1.5.6.8.

Params:

• data: string|bytes - input compressed data block, length limited by 2Gb by Python API

Aliases:

• decompress(…),

• uncompress(…),

• loads(…),

• decode(…) since: 1.5.6.2

Since: 0.1

ZSTD_check (data): int

Function, checks if input is zstd compressed data block, and returns: 1 if yes, 0 if no or 2 if it is a stream data.

Support compressed data with multiple/concatenated frames (blocks) .

Params:

• data: string|bytes - input compressed data block, length limited by 2Gb by Python API

Aliases:

• check(…),

• verify(…) since: 1.5.6.3

Since: 1.5.6.2

version (): string|bytes

Returns this module doted version string.

The first three digits are folow libzstd version. Fourth digit - module revision number for that version.

Since: 1.3.4.3

ZSTD_version (): string|bytes

Returns ZSTD library doted version string, with which are compiled.

Since: 1.3.4.3

Aliases:

• ZSTD_version_compiled (): string|bytes Since: 1.5.7.3

ZSTD_version_loaded (): string|bytes

Returns ZSTD library doted version string, with which currently loaded.

Since: 1.5.7.3

ZSTD_version_number (): int

Returns ZSTD library version, compiled with, in format: MAJOR*100*100 + MINOR*100 + RELEASE.

Since: 1.3.4.3

ZSTD_threads_count (): int

Returns ZSTD determined CPU cores count. Better to get this nuber as early as possible befor compression starts.

Since: 1.5.4.1

ZSTD_max_threads_count (): int

Returns ZSTD library determined maximum working threads count.

Since: 1.5.4.1

ZSTD_max_compression_level (): int

Returns ZSTD library determined maximum number of compression level .

Since: 1.5.6.3

ZSTD_min_compression_level (): int

Returns ZSTD library determined minimum number of compression level .

Since: 1.5.6.3

ZSTD_default_compression_level (): int

Returns ZSTD library determined default number of compression level .

Since: 1.5.7.1

ZSTD_external (): int

Returns 0 of 1 if ZSTD library linked as external.

Since: 1.5.0.2

ZSTD_legacy_support (): int

Returns 0 of 1 if ZSTD library built with legacy formats support.

Since: 1.5.6.3

ZSTD_with_threads (): int

Returns 0 of 1 if bundled ZSTD library build with threads support.

Since: 1.5.6.2

ZSTD_with_asm (): int

Returns 0 of 1 if bundled ZSTD library build with asm optimizations.

Since: 1.5.6.2

ZSTD_is_debug_enable (): int

Returns 0 of 1 if module built with debug output.

Since: 1.5.7.1

ZSTD_is_debug_notice_enable (): int

Returns 0 of 1 if module built with debug output - notice level.

Since: 1.5.7.1

ZSTD_is_debug_info_enable (): int

Returns 0 of 1 if module built with debug output - info level.

Since: 1.5.7.1

ZSTD_is_debug_error_enable (): int

Returns 0 of 1 if module built with debug output - error level.

Since: 1.5.7.1

Removed

ZSTD_compress_old (data[, level]): string|bytes

Function, compress input data block, return compressed block, or raises Error.

DEPRECATED: Returns not compatible with ZSTD block header

REMOVED: since 1.5.0

Params:

• data: string|bytes - input data block, length limited by 2Gb by Python API

• level: int - compression level, ultra-fast levels from -5 (ultra) to -1 (fast) available since zstd-1.3.4, and from 1 (fast) to 22 (slowest), 0 or unset - means default (3). Default - 3.

Since: 1.0.0.99.1

ZSTD_uncompress_old (data): string|bytes

Function, decompress input compressed data block, return decompressed block, or raises Error.

DEPRECATED: Accepts data with not compatible with ZSTD block header

REMOVED: since 1.5.0

Params:

• data: string|bytes - input compressed data block, length limited by 2Gb by Python API

Since: 1.0.0.99.1

Use

Module has simple API:

>>> import zstd
>>> dir(zstd)
['Error', 'ZSTD_compress', 'ZSTD_external', 'ZSTD_uncompress', 'ZSTD_version', 'ZSTD_version_number', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', 'compress', 'decompress', 'dumps', 'loads', 'uncompress', 'version']
>>> zstd.version()
'1.5.1.0'
>>> zstd.ZSTD_version()
'1.5.1'
>>> zstd.ZSTD_version_number()
10501
>>> zstd.ZSTD_external()
0

In python2

>>> data = "123456qwert"

In python3 use bytes

>>> data = b"123456qwert"

>>> cdata = zstd.compress(data, 1)
>>> data == zstd.decompress(cdata)
True
>>> cdata_mt = zstd.compress(data, 1, 4)
>>> cdata == cdata_mt
True
>>> data == zstd.decompress(cdata_mt)
True