This library provides a simple Python interface for implementing erasure codes and is known to work with Python 3.10 through 3.13. To obtain the best possible performance, the library utilizes liberasurecode, which is a C based erasure code library.
PyECLib supports a variety of Erasure Coding backends including the standard Reed-Solomon implementations provided by Jerasure [1], liberasurecode [3], Intel’s ISA-L [4] and Phazr.IO’s libphazr. It also provides support for a flat XOR-based encoder and decoder (part of liberasurecode) - a class of HD Combination Codes based on “Flat XOR-based erasure codes in storage systems: Constructions, efficient recovery, and tradeoffs” in IEEE MSST 2010[2]). These codes are well-suited to archival use-cases, have a simple construction and require a minimum number of participating disks during single-disk reconstruction (think XOR-based LRC code).
Installation (PyPI)
$ pip install pyeclibBinary wheels on PyPI are provided for x86_64 and aarch64, with vendored versions of liberasurecode and ISA-L included.
Installation (System Libraries)
Install pre-requisites:
Python 3.10+ (including development packages), argparse, setuptools
liberasurecode v1.4.0 or greater [3]
Erasure code backend libraries, gf-complete and Jerasure [1], ISA-L [4], etc.
Install dependencies:
Debian/Ubuntu hosts:
$ sudo apt-get install build-essential python-dev python-pip liberasurecode-dev $ sudo pip install -U bindep -r test-requirements.txt
RHEL/CentOS hosts:
$ sudo yum install -y redhat-lsb python2-pip python-devel liberasurecode-devel $ sudo pip install -U bindep -r test-requirements.txt $ tools/test-setup.sh
If you want to confirm all dependency packages installed successfully, try:
$ sudo bindep -f bindep.txt
For CentOS, make sure to install the latest Openstack Cloud SIG repo to be able to install the latest available version of liberasurecode-devel.
Install PyECLib:
$ sudo python setup.py install
Run test suite included:
$ ./.unittests
If the test suite fails because it cannot find any of the shared libraries, then you probably need to add /usr/local/lib to the path searched when loading libraries. The best way to do this (on Linux) is to add ‘/usr/local/lib’ to:
/etc/ld.so.conf
and then make sure to run:
$ sudo ldconfig
Getting started
Examples of using PyECLib are provided in the “tools” directory:
Command-line encoder:
tools/pyeclib_encode.py
Command-line decoder:
tools/pyeclib_decode.py
Utility to determine what is needed to reconstruct missing fragments:
tools/pyeclib_fragments_needed.py
A configuration utility to help compare available EC schemes in terms of performance and redundancy:
tools/pyeclib_conf_tool.py
PyEClib initialization:
ec_driver = ECDriver(k=<num_encoded_data_fragments>,
m=<num_encoded_parity_fragments>,
ec_type=<ec_scheme>))
Supported ec_type values:
liberasurecode_rs_vand => Vandermonde Reed-Solomon encoding, software-only backend implemented by liberasurecode [3]
jerasure_rs_vand => Vandermonde Reed-Solomon encoding, based on Jerasure [1]
jerasure_rs_cauchy => Cauchy Reed-Solomon encoding (Jerasure variant), based on Jerasure [1]
flat_xor_hd_3, flat_xor_hd_4 => Flat-XOR based HD combination codes, liberasurecode [3]
isa_l_rs_vand => Intel Storage Acceleration Library (ISA-L) - SIMD accelerated Erasure Coding backends [4]
isa_l_rs_cauchy => Cauchy Reed-Solomon encoding (ISA-L variant) [4]
shss => NTT Lab Japan’s Erasure Coding Library [5]
libphazr => Phazr.IO’s erasure code library with built-in privacy [6]
Code Maintenance
This library is currently mainly maintained by the Openstack Swift community. For questions or any other help, come ask in #openstack-swift on OFTC.
References
[1] Jerasure, C library that supports erasure coding in storage applications, http://jerasure.org
[2] Greenan, Kevin M et al, “Flat XOR-based erasure codes in storage systems”, https://web.archive.org/web/20161001210233/https://www.kaymgee.com/Kevin_Greenan/Publications_files/greenan-msst10.pdf
[3] liberasurecode, C API abstraction layer for erasure coding backends, https://opendev.org/openstack/liberasurecode
[4] Intel(R) Storage Acceleration Library (Open Source Version), https://01.org/intel%C2%AE-storage-acceleration-library-open-source-version
[5] Kota Tsuyuzaki <tsuyuzaki.kota@lab.ntt.co.jp>, “NTT SHSS Erasure Coding backend”
[6] Jim Cheung <support@phazr.io>, “Phazr.IO libphazr erasure code backend with built-in privacy”
5 maintainers
Release files for pyeclib 1.8.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| pyeclib-1.8.0.tar.gz | 48.4 kB | Details |
Built distributions (wheels)
Total release size: 4.4 MB
Release files / pyeclib-1.8.0.tar.gz
| Download URL | pyeclib-1.8.0.tar.gz |
|---|---|
| Size | 48.4 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
a961675422953e9c54338dcea9f3148bc0e82499efb0ae5acba7373a9d26a627
|
|
BLAKE2b-256 checksum How to use checksums |
4348594a6c3ec20cfb37581b028836db55b4834fc7259d01b22069fee7150ab2
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.14
|
Release files / pyeclib-1.8.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
| Download URL | pyeclib-1.8.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl |
|---|---|
| Size | 569.6 kB |
| Tags | CPython 3.14 CPython 3.14 free-threading Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
447efb7167a368f477dd1d12c73b57c72be1aa4dffc65e2523f646eff90574e0
|
|
BLAKE2b-256 checksum How to use checksums |
43a0d016d612114d14f7a42775b3166bfdaf5177f40e99932548e97eaff6bd8c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.14
|
Release files / pyeclib-1.8.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
| Download URL | pyeclib-1.8.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl |
|---|---|
| Size | 533.8 kB |
| Tags | CPython 3.14 CPython 3.14 free-threading Linux glibc 2.17+ ARM64 |
|
SHA-256 checksum How to use checksums |
98446d408e1e9b4e2350238fd84b4f698305b9885454797a74ee645c9225231d
|
|
BLAKE2b-256 checksum How to use checksums |
48efdc4562af563a63b849d2914836cc9b9b235daa9c228ed490628aaac61e16
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.14
|
Release files / pyeclib-1.8.0-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
| Download URL | pyeclib-1.8.0-cp313-cp313t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl |
|---|---|
| Size | 569.6 kB |
| Tags | CPython 3.13 CPython 3.13 free-threading Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
742b84134ab59b3f70ed0d74537b629ba7938858c6ff5bcae2d0021abffabbe4
|
|
BLAKE2b-256 checksum How to use checksums |
3e558f07fdd7c9bc007f7084287e447b9e242193fdb6b8ce649c217e6b7140c6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.14
|
Release files / pyeclib-1.8.0-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
| Download URL | pyeclib-1.8.0-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl |
|---|---|
| Size | 533.8 kB |
| Tags | CPython 3.13 CPython 3.13 free-threading Linux glibc 2.17+ ARM64 |
|
SHA-256 checksum How to use checksums |
bbe92fab366919bf579946c1d9d40ce25391662a2d4f05e941a64b9d56c2d4fe
|
|
BLAKE2b-256 checksum How to use checksums |
9b1deb4102c0f8b5b3ef020c8b87de8198812a45577f6ccee61ba87cdf5d3d40
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.14
|
Release files / pyeclib-1.8.0-cp310-abi3-musllinux_1_2_x86_64.whl
| Download URL | pyeclib-1.8.0-cp310-abi3-musllinux_1_2_x86_64.whl |
|---|---|
| Size | 555.6 kB |
| Tags | CPython 3.10 Linux musl 1.2+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
fa1eccaa152b3e18fe846ef3ff5371c81a3c4ea776ecc566c86d4d5068a48003
|
|
BLAKE2b-256 checksum How to use checksums |
8c78467028fbb9ea6c39bb54ffbc031fedbc281657b232a4d929fea42e2066f4
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.14
|
Release files / pyeclib-1.8.0-cp310-abi3-musllinux_1_2_aarch64.whl
| Download URL | pyeclib-1.8.0-cp310-abi3-musllinux_1_2_aarch64.whl |
|---|---|
| Size | 519.8 kB |
| Tags | CPython 3.10 Linux musl 1.2+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
44f391e7f4973a7cee132d8c94a3d13ff16a3e4dcbe7ee03abc9f53e890d1e82
|
|
BLAKE2b-256 checksum How to use checksums |
1b763054075334e095f4db84f6cd53fd2d02d3ffc19482331bf44b7f48fabf91
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.14
|
Release files / pyeclib-1.8.0-cp310-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl
| Download URL | pyeclib-1.8.0-cp310-abi3-manylinux2014_x86_64.manylinux_2_17_x86_64.whl |
|---|---|
| Size | 568.9 kB |
| Tags | CPython 3.10 Linux glibc 2.17+ x86-64 abi3 |
|
SHA-256 checksum How to use checksums |
2036956a8b131a56baa86bd58697ae8efbdda0a8d52a66ef25fdda87e1b1229a
|
|
BLAKE2b-256 checksum How to use checksums |
aa29a1f77f08c46017c67f908c32e6c2614e4dae5424b833fcd7d0251bd8e361
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.14
|
Release files / pyeclib-1.8.0-cp310-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl
| Download URL | pyeclib-1.8.0-cp310-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl |
|---|---|
| Size | 533.2 kB |
| Tags | CPython 3.10 Linux glibc 2.17+ ARM64 abi3 |
|
SHA-256 checksum How to use checksums |
b4ecef839ff779697a86877fdf562a3ff131fa21ba8c272c0b75a23215620215
|
|
BLAKE2b-256 checksum How to use checksums |
d1b86e53c3145ac1bdcdf8142d89d0765ff6ab6ab6f8deafcd1ef1032f559369
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.2.0 CPython/3.11.14
|
