This library provides a simple Python interface for implementing erasure codes and is known to work with Python 3.10 through 3.13. To obtain the best possible performance, the library utilizes liberasurecode, which is a C based erasure code library.

PyECLib supports a variety of Erasure Coding backends including the standard Reed-Solomon implementations provided by Jerasure [1], liberasurecode [3], Intel’s ISA-L [4] and Phazr.IO’s libphazr. It also provides support for a flat XOR-based encoder and decoder (part of liberasurecode) - a class of HD Combination Codes based on “Flat XOR-based erasure codes in storage systems: Constructions, efficient recovery, and tradeoffs” in IEEE MSST 2010[2]). These codes are well-suited to archival use-cases, have a simple construction and require a minimum number of participating disks during single-disk reconstruction (think XOR-based LRC code).

Installation (PyPI)

$ pip install pyeclib

Binary wheels on PyPI are provided for x86_64 and aarch64, with vendored versions of liberasurecode and ISA-L included.

Installation (System Libraries)

Install pre-requisites:

  • Python 3.10+ (including development packages), argparse, setuptools

  • liberasurecode v1.4.0 or greater [3]

  • Erasure code backend libraries, gf-complete and Jerasure [1], ISA-L [4], etc.

Install dependencies:

Debian/Ubuntu hosts:

$ sudo apt-get install build-essential python-dev python-pip liberasurecode-dev
$ sudo pip install -U bindep -r test-requirements.txt

RHEL/CentOS hosts:

$ sudo yum install -y redhat-lsb python2-pip python-devel liberasurecode-devel
$ sudo pip install -U bindep -r test-requirements.txt
$ tools/test-setup.sh

If you want to confirm all dependency packages installed successfully, try:

$ sudo bindep -f bindep.txt

For CentOS, make sure to install the latest Openstack Cloud SIG repo to be able to install the latest available version of liberasurecode-devel.

Install PyECLib:

$ sudo python setup.py install

Run test suite included:

$ ./.unittests

If the test suite fails because it cannot find any of the shared libraries, then you probably need to add /usr/local/lib to the path searched when loading libraries. The best way to do this (on Linux) is to add ‘/usr/local/lib’ to:

/etc/ld.so.conf

and then make sure to run:

$ sudo ldconfig

Getting started

Examples of using PyECLib are provided in the “tools” directory:

Command-line encoder:

tools/pyeclib_encode.py

Command-line decoder:

tools/pyeclib_decode.py

Utility to determine what is needed to reconstruct missing fragments:

tools/pyeclib_fragments_needed.py

A configuration utility to help compare available EC schemes in terms of performance and redundancy:

tools/pyeclib_conf_tool.py

PyEClib initialization:

ec_driver = ECDriver(k=<num_encoded_data_fragments>,
                     m=<num_encoded_parity_fragments>,
                     ec_type=<ec_scheme>))

Supported ec_type values:

  • liberasurecode_rs_vand => Vandermonde Reed-Solomon encoding, software-only backend implemented by liberasurecode [3]

  • jerasure_rs_vand => Vandermonde Reed-Solomon encoding, based on Jerasure [1]

  • jerasure_rs_cauchy => Cauchy Reed-Solomon encoding (Jerasure variant), based on Jerasure [1]

  • flat_xor_hd_3, flat_xor_hd_4 => Flat-XOR based HD combination codes, liberasurecode [3]

  • isa_l_rs_vand => Intel Storage Acceleration Library (ISA-L) - SIMD accelerated Erasure Coding backends [4]

  • isa_l_rs_cauchy => Cauchy Reed-Solomon encoding (ISA-L variant) [4]

  • shss => NTT Lab Japan’s Erasure Coding Library [5]

  • libphazr => Phazr.IO’s erasure code library with built-in privacy [6]

Code Maintenance

This library is currently mainly maintained by the Openstack Swift community. For questions or any other help, come ask in #openstack-swift on OFTC.

References

[1] Jerasure, C library that supports erasure coding in storage applications, http://jerasure.org

[2] Greenan, Kevin M et al, “Flat XOR-based erasure codes in storage systems”, https://web.archive.org/web/20161001210233/https://www.kaymgee.com/Kevin_Greenan/Publications_files/greenan-msst10.pdf

[3] liberasurecode, C API abstraction layer for erasure coding backends, https://opendev.org/openstack/liberasurecode

[4] Intel(R) Storage Acceleration Library (Open Source Version), https://01.org/intel%C2%AE-storage-acceleration-library-open-source-version

[5] Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>, “NTT SHSS Erasure Coding backend”

[6] Jim Cheung <support@phazr.io>, “Phazr.IO libphazr erasure code backend with built-in privacy”

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

5 maintainers

PyPI data
Data sourced directly from PyPI's database.
Avatar for clayg from gravatar.com clayg Avatar for kmgreen2 from gravatar.com kmgreen2 Avatar for openstackci from gravatar.com openstackci
+2 more Avatar for thiagodasilva from gravatar.com thiagodasilva Avatar for tsg from gravatar.com tsg

Credits

Author: Kevin Greenan

Maintainer: Tim Burke

License expression

BSD-2-Clause
View SPDX License List

Requires

Python >=3.10

Classifiers

Report project as malware

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pyeclib-1.8.0.tar.gz (48.4 kB view details)

Uploaded Source

Built Distributions

If you're not sure about the file name format, learn more about wheel file names.

pyeclib-1.8.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (569.6 kB view details)

Uploaded CPython 3.14tmanylinux: glibc 2.17+ x86-64

pyeclib-1.8.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (533.8 kB view details)

Uploaded CPython 3.14tmanylinux: glibc 2.17+ ARM64

pyeclib-1.8.0-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (569.6 kB view details)

Uploaded CPython 3.13tmanylinux: glibc 2.17+ x86-64

pyeclib-1.8.0-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (533.8 kB view details)

Uploaded CPython 3.13tmanylinux: glibc 2.17+ ARM64

pyeclib-1.8.0-cp310-abi3-musllinux_1_2_x86_64.whl (555.6 kB view details)

Uploaded CPython 3.10+musllinux: musl 1.2+ x86-64

pyeclib-1.8.0-cp310-abi3-musllinux_1_2_aarch64.whl (519.8 kB view details)

Uploaded CPython 3.10+musllinux: musl 1.2+ ARM64

pyeclib-1.8.0-cp310-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (568.9 kB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ x86-64

pyeclib-1.8.0-cp310-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl (533.2 kB view details)

Uploaded CPython 3.10+manylinux: glibc 2.17+ ARM64

File details

Details for the file pyeclib-1.8.0.tar.gz.

File metadata

  • Download URL: pyeclib-1.8.0.tar.gz
  • Upload date:
  • Size: 48.4 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.2.0 CPython/3.11.14

File hashes

Hashes for pyeclib-1.8.0.tar.gz
Algorithm Hash digest
SHA256 a961675422953e9c54338dcea9f3148bc0e82499efb0ae5acba7373a9d26a627
MD5 ac6a2e56513e58ca8db6bf772893d299
BLAKE2b-256 4348594a6c3ec20cfb37581b028836db55b4834fc7259d01b22069fee7150ab2

See more details on using hashes here.

File details

Details for the file pyeclib-1.8.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.

File metadata

File hashes

Hashes for pyeclib-1.8.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Algorithm Hash digest
SHA256 447efb7167a368f477dd1d12c73b57c72be1aa4dffc65e2523f646eff90574e0
MD5 a982a1e9add02132428e551ae6adc218
BLAKE2b-256 43a0d016d612114d14f7a42775b3166bfdaf5177f40e99932548e97eaff6bd8c

See more details on using hashes here.

File details

Details for the file pyeclib-1.8.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl.

File metadata

File hashes

Hashes for pyeclib-1.8.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
Algorithm Hash digest
SHA256 98446d408e1e9b4e2350238fd84b4f698305b9885454797a74ee645c9225231d
MD5 6d42ec45f06becd26ae07e979a7bbb1b
BLAKE2b-256 48efdc4562af563a63b849d2914836cc9b9b235daa9c228ed490628aaac61e16

See more details on using hashes here.

File details

Details for the file pyeclib-1.8.0-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.

File metadata

File hashes

Hashes for pyeclib-1.8.0-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Algorithm Hash digest
SHA256 742b84134ab59b3f70ed0d74537b629ba7938858c6ff5bcae2d0021abffabbe4
MD5 bee498fe8c90fe7643bfbccc8c743455
BLAKE2b-256 3e558f07fdd7c9bc007f7084287e447b9e242193fdb6b8ce649c217e6b7140c6

See more details on using hashes here.

File details

Details for the file pyeclib-1.8.0-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl.

File metadata

File hashes

Hashes for pyeclib-1.8.0-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
Algorithm Hash digest
SHA256 bbe92fab366919bf579946c1d9d40ce25391662a2d4f05e941a64b9d56c2d4fe
MD5 7e9d8945dde88c5271bb34f3775b923e
BLAKE2b-256 9b1deb4102c0f8b5b3ef020c8b87de8198812a45577f6ccee61ba87cdf5d3d40

See more details on using hashes here.

File details

Details for the file pyeclib-1.8.0-cp310-abi3-musllinux_1_2_x86_64.whl.

File metadata

File hashes

Hashes for pyeclib-1.8.0-cp310-abi3-musllinux_1_2_x86_64.whl
Algorithm Hash digest
SHA256 fa1eccaa152b3e18fe846ef3ff5371c81a3c4ea776ecc566c86d4d5068a48003
MD5 2f3dbfbb78451efcc3d4c8420c5dd069
BLAKE2b-256 8c78467028fbb9ea6c39bb54ffbc031fedbc281657b232a4d929fea42e2066f4

See more details on using hashes here.

File details

Details for the file pyeclib-1.8.0-cp310-abi3-musllinux_1_2_aarch64.whl.

File metadata

File hashes

Hashes for pyeclib-1.8.0-cp310-abi3-musllinux_1_2_aarch64.whl
Algorithm Hash digest
SHA256 44f391e7f4973a7cee132d8c94a3d13ff16a3e4dcbe7ee03abc9f53e890d1e82
MD5 f027fa50ba8dc7e3f14eb93615a6bb61
BLAKE2b-256 1b763054075334e095f4db84f6cd53fd2d02d3ffc19482331bf44b7f48fabf91

See more details on using hashes here.

File details

Details for the file pyeclib-1.8.0-cp310-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.

File metadata

File hashes

Hashes for pyeclib-1.8.0-cp310-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
Algorithm Hash digest
SHA256 2036956a8b131a56baa86bd58697ae8efbdda0a8d52a66ef25fdda87e1b1229a
MD5 ea8651ffe45be82bdca98e5ba4535047
BLAKE2b-256 aa29a1f77f08c46017c67f908c32e6c2614e4dae5424b833fcd7d0251bd8e361

See more details on using hashes here.

File details

Details for the file pyeclib-1.8.0-cp310-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl.

File metadata

File hashes

Hashes for pyeclib-1.8.0-cp310-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
Algorithm Hash digest
SHA256 b4ecef839ff779697a86877fdf562a3ff131fa21ba8c272c0b75a23215620215
MD5 24502d3cf4a4f4a66bb7568d9298e5e0
BLAKE2b-256 d1b86e53c3145ac1bdcdf8142d89d0765ff6ab6ab6f8deafcd1ef1032f559369

See more details on using hashes here.

Release history Release notifications | RSS feed

This release

1.8.0 This release

9 files

1.7.0

5 files

1.6.4

7 files

1.6.3

7 files

1.6.2

7 files

Pre-release

1.6.2rc1

7 files

1.6.1

1 file

1.6.0

1 file

1.5.0

1 file

1.4.0

1 file

1.3.1

1 file

1.2.0

1 file

1.1.1

1 file

1.1.0

1 file

1.0.9

1 file

1.0.7

1 file

1.0.6

1 file

1.0.4

1 file

1.0.3

1 file

1.0.2

1 file

1.0.1

1 file

Pre-release

1.0c2

1 file

Pre-release

1.0c1

1 file

0.9.8

1 file

0.9.7

1 file

0.9.6

1 file

0.9.5

1 file

0.9.4

1 file

0.9.2

1 file

0.9.1

1 file

0.2.4

1 file

0.2.3

1 file

0.2.2

1 file

0.2.1

1 file

0.2.0

1 file

0.1.17

1 file

0.1.16

1 file

0.1.15

1 file

0.1.14

1 file

0.1.12

1 file

0.1.11

1 file

0.1.10

1 file

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page