This library provides a simple Python interface for implementing erasure codes. To obtain the best possible performance, the underlying erasure code algorithms are written in C.
This is v0.9 of PyECLib. This library provides a simple Python interface for implementing erasure codes and is known to work with Python v2.6, 2.7 and 3.x.
To obtain the best possible performance, the underlying erasure code algorithms are written in C. Please let us know if you have any other issues building or installing (email: firstname.lastname@example.org or email@example.com).
This library makes use of Jerasure for Reed-Solomon and provides its own flat XOR-based erasure code encoder and decoder. Currently, it implements a specific class of HD Combination Codes (see “Flat XOR-based erasure codes in storage systems: Constructions, efficient recovery, and tradeoffs” in IEEE MSST 2010). These codes are well-suited to archival use-cases, have a simple construction and require a minimum number of participating disks during single-disk reconstruction (think XOR-based LRC code).
The XOR-based codes make use of 128-bit registers using the Intel SIMD extensions. We are currently working on incorporating GF-Complete into Jerasure. Version 1.0 of this library will link to a new version of Jerasure that uses the Intel SIMD instruction set for Vandermonde Reed-Solomon encoding (see GF-Complete 1.0: http://web.eecs.utk.edu/~plank/plank/papers/CS-13-703.html and “Screaming fast Galois Field arithmetic using Intel SIMD instructions” in USENIX FAST 2013).
Examples of using this library are provided in ./tools:
Utility to determine what is needed to reconstruct missing fragments:pyeclib_fragments_needed.py
ec_driver = ECDriver("pyeclib.core.ECPyECLibDriver", k=<num_encoded_data_fragments>, m=<num_encoded_parity_fragments>, ec_type=<ec_scheme>))
- rs_vand => Vandermonde Reed-Solomon encoding
- flat_xor_3, flat_xor_4 => Flat-XOR based HD combination codes
A configuration utility is provided to help compare available EC schemes in terms of performance and redundancy:: tools/pyeclib_conf_tool.py
The Python API supports the following functions:
Encode N bytes of a data object into k (data) + m (parity) fragments:
def encode(self, data_bytes) input: data_bytes - input data object (bytes) returns: list of fragments (bytes)
Decode between k and k+m fragments into original object:
def decode(self, fragment_payloads) input: list of fragment_payloads (bytes) returns: decoded object (bytes)
Note: bytes is a synonym to str in Python 2.6, 2.7. In Python 3.x, bytes and str types are non-interchangeable and care needs to be taken when handling input to and output from the encode() and decode() routines.
Reconstruct “missing_fragment_indexes” using “available_fragment_payloads”:
def reconstruct(self, available_fragment_payloads, missing_fragment_indexes)
Fragments needed for EC Reconstruct
Return the indexes of fragments needed to reconstruct “missing_fragment_indexes”:
def fragments_needed(self, missing_fragment_indexes)
Get EC Metadata
Return an opaque buffer known by the underlying library:
def get_metadata(self, fragment)
Verify EC Stripe Consistency
Use opaque buffers from get_metadata() to verify a the consistency of a stripe:
def verify_stripe_metadata(self, fragment_metadata_list)
Get EC Segment Info
Return a dict with the keys - segment_size, last_segment_size, fragment_size, last_fragment_size and num_segments:
def get_segment_info(self, data_len, segment_size)
Standard stuff to install:Python 2.6, 2.7 or 3.x (including development packages) and argparse.
This package includes c_eclib, which contains a few C libraries used by PyECLib. These will be built when running the PyECLib install (see below).
Install PyECLib:$ sudo python setup.py install
Run the test suite:$ (cd test && \ python run_tests.py && \ ./ec_pyeclib_file_test.sh)
If all of this works, then you should be good to go. If not, send us an email!
If the test suite fails because it cannot find any of the shared libraries, then you probably need to add /usr/local/lib to the path searched when loading libraries. The best way to do this (on Linux) is to add ‘/usr/local/lib’ to:/etc/ld.so.conf
and then run:$ ldconfig