Skip to main content

Efficient binary storage of ML models

Project description

BinPickle - efficient binary pickled data

PyPI version Test and Build codecov

This package uses the new Pickle Protocol 5 in Python 3.8 (or its pickle5 backport) to efficiently serialize large objects, particularly from scientific Python packages, to an on-disk format. This format is designed to support two use cases:

  1. Serializing data-intensive statistical models in a memory-mappable format so multiple processes can share the same (read-only) model memory.
  2. Serializing data-intensive statistical models with good compression for long-term storage and cross-machine transportation.

BinPickle does this by using Pickle 5's out-of-band buffer serialization support to write buffers uncompressed and page-aligned for memory mapping (use case 1) or with per-buffer efficient compression with libraries like Blosc (use case 2).

Format Stability

We do not yet guarantee the stability of the BinPickle format. We will avoid gratuitous changes, but BinPickle 1.0 will be the first with a stability guarantee.

Acknowledgements

This material is based upon work supported by the National Science Foundation under Grant No. IIS 17-51278. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation. This page has not been approved by Boise State University and does not reflect official university positions.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for binpickle, version 0.3.3
Filename, size File type Python version Upload date Hashes
Filename, size binpickle-0.3.3-py3-none-any.whl (14.2 kB) File type Wheel Python version py3 Upload date Hashes View
Filename, size binpickle-0.3.3.tar.gz (15.8 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page