Skip to main content
Python Software Foundation 20th Year Anniversary Fundraiser  Donate today!

Resumable hashlib: a picklable interface to CPython's OpenSSL-based hashlib standard library

Project description

Rehash is a resumable interface to the OpenSSL-based hashers (message digest objects) in the CPython hashlib standard library. Rehash provides hashers that can be pickled, persisted and reconstituted from their repr(), and otherwise serialized. The rest of the Rehash API is identical to hashlib.

Rehash hashers can be used to checkpoint and restore progress when hashing large byte streams:

import pickle, rehash
hasher = rehash.sha256(b"foo")
state = pickle.dumps(hasher)

hasher2 = pickle.loads(state)
hasher2.update(b"bar")

assert hasher2.hexdigest() == rehash.sha256(b"foobar").hexdigest()

Installation

pip install rehash

Applications

Rehash is useful in any situation when your VM is short-lived or preemptable, and the object you’re hashing is huge. For example, Rehash can be used to hand off the hashing state of large objects between AWS Lambda functions or Google Cloud Functions, which have runtime limits of 15 and 9 minutes, respectively.

Non-openssl hashers

sha3 and blake2 hash algorithms in Python 3.6 are not OpenSSL-based and not supported by rehash.

PyPy

PyPy uses its own hasher implementations. Those are not serializable using rehash.

Security note

By default, rehash objects present themselves with a repr() that exposes their internal state. This allows one to resume the hashing from the point where it stopped. If exposed through an untrusted channel under specific conditions, this could potentially allow an attacker to use an extension attack. If you are unsure about the implications of this, set rehash.opaque_repr = True after importing rehash.

License

Licensed under the terms of the Apache License, Version 2.0.

https://travis-ci.org/kislyuk/rehash.png https://codecov.io/github/kislyuk/rehash/coverage.svg?branch=master https://img.shields.io/pypi/v/rehash.svg https://img.shields.io/pypi/l/rehash.svg https://readthedocs.org/projects/rehash/badge/?version=latest

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for rehash, version 1.0.0
Filename, size File type Python version Upload date Hashes
Filename, size rehash-1.0.0-py2.py3-none-any.whl (8.7 kB) File type Wheel Python version py2.py3 Upload date Hashes View
Filename, size rehash-1.0.0.tar.gz (5.0 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page