Skip to main content

This library provides a simple Python interface for implementing erasure codes. To obtain the best possible performance, the underlying erasure code algorithms are written in C.

Project description

This is v0.1 of PyECLib. This library provides a simple Python interface for implementing erasure codes. To obtain the best possible performance, the underlying erasure code algorithms are written in C. Please let me know if you have any other issues building/installing (email: kmgreen2@gmail.com).

This library makes use of Jerasure for Reed-Solomon and provides its own flat XOR-based erasure code encoder and decoder. Currently, it implements a specific class of HD Combination Codes (see “Flat XOR-based erasure codes in storage systems: Constructions, efficient recovery, and tradeoffs” in IEEE MSST 2010). These codes are well-suited to archival use-cases, have a simple construction and require a minimum number of participating disks during single-disk reconstruction (think XOR-based LRC code).

The XOR-based codes make use of 128-bit registers using the Intel SIMD extensions. We are currently working on incorporating GF-Complete into Jerasure. Version 1.0 of this library will link to a new version of Jerasure that uses the Intel SIMD instruction set for Vandermonde Reed-Solomon encoding (see GF-Complete 1.0: http://web.eecs.utk.edu/~plank/plank/papers/CS-13-703.html and “Screaming fast Galois Field arithmetic using Intel SIMD instructions” in USENIX FAST 2013).

Examples of using this library are provided in ./tools:

Command-line encoder: ec_pyeclib_encode.py

Command-line decoder: ec_pyeclib_decode.py

Utility to determine what is needed to reconstruct missing fragments: pyeclib_fragments_needed.py

The main Python interface only contains 6 functions:

Encode N bytes into k+m fragments (returns list of fragments):

def encode(self, bytes)

Decode between k and k+m fragments into a string (returns a string):

def decode(self, fragment_payloads)

Reconstruct “missing_fragment_indexes” using “available_fragment_payloads”:

def reconstruct(self, available_fragment_payloads, missing_fragment_indexes)

Return the indexes of fragments needed to reconstruct “missing_fragment_indexes”:

def fragments_needed(self, missing_fragment_indexes)

Return an opaque buffer known by the underlying library:

def get_metadata(self, fragment)

Use opaque buffers from get_metadata() to verify a the consistency of a stripe:

def verify_stripe_metadata(self, fragment_metadata_list)

Quick Start:

Standard stuff to install: Python 2.7 (including devel), argparse

If you do not have Jerasure installed. Install it from the unofficial GitHub repo linked from here (setup.py assumes it will be in /usr/local):

https://bitbucket.org/kmgreen2/jerasure-kmg

This version of Jerasure is a development version that includes SIMD support using GF-Complete. You will also need to install GF-Complete:

https://bitbucket.org/jimplank/gf-complete

GF-Complete does not have an ‘install’ target, so either put an install target into the makefile, or

1.) cp gf_complete.a /usr/local/lib/libgf_complete.a 2.) cp gf_complete.h /usr/local/include

Alternatively, you can install my fork of gf-complete, which includes an install:

https://bitbucket.org/kmgreen2/gf-complete-kmg

Install PyECLib:

sudo python setup.py install

Run the test suite:

cd test; python run_tests.py; ./ec_pyeclib_file_test.sh; cd ..

If all of this works, then you should be good to go. If not, send me an email!

If the test suite fails because it cannot find any of the shared libraries, then you probably need to add /usr/local/lib to the path searched when loading libraries. The best way to do this (on Linux) is to add ‘/usr/local/lib’ to

/etc/ld.so.conf

and then run:

ldconfig

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

PyECLib-0.1.10.tar.gz (7.0 MB view details)

Uploaded Source

File details

Details for the file PyECLib-0.1.10.tar.gz.

File metadata

  • Download URL: PyECLib-0.1.10.tar.gz
  • Upload date:
  • Size: 7.0 MB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for PyECLib-0.1.10.tar.gz
Algorithm Hash digest
SHA256 a78206822a07a55da457fa70987123dc7ad96ae6f21abdd9f0b318a8cd797507
MD5 ed6cd88f2654b487e6dcb7c4533c7175
BLAKE2b-256 4a3f8e5c2f219a8c7a896164f3bbd64008b46d410be46ad37284ae185db327b4

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page