A Python wrapper for the extremely fast Blosc2 compression library
- Author:
The Blosc development team
- Contact:
- Github:
- Actions:
- PyPi:
- NumFOCUS:
- Code of Conduct:
What it is
C-Blosc2 is the new major version of C-Blosc, and is backward compatible with both the C-Blosc1 API and its in-memory format. Python-Blosc2 is a Python package that wraps C-Blosc2, the newest version of the Blosc compressor.
Currently Python-Blosc2 already reproduces the API of Python-Blosc, so it can be used as a drop-in replacement. However, there are a few exceptions for a full compatibility.
In addition, Python-Blosc2 aims to leverage the new C-Blosc2 API so as to support super-chunks, multi-dimensional arrays (NDArray), serialization and other bells and whistles introduced in C-Blosc2. Although this is always and endless process, we have already catch up with most of the C-Blosc2 API capabilities.
Note: Python-Blosc2 is meant to be backward compatible with Python-Blosc data. That means that it can read data generated with Python-Blosc, but the opposite is not true (i.e. there is no forward compatibility).
SChunk: a 64-bit compressed store
SChunk is the simple data container that handles setting, expanding and getting data and metadata. Contrarily to chunks, a super-chunk can update and resize the data that it contains, supports user metadata, and it does not have the 2 GB storage limitation.
Additionally, you can convert a SChunk into a contiguous, serialized buffer (aka cframe) and vice-versa; as a bonus, the serialization/deserialization process also works with NumPy arrays and PyTorch/TensorFlow tensors at a blazing speed:
while reaching excellent compression ratios:
Also, if you are a Mac M1/M2 owner, make you a favor and use its native arm64 arch (yes, we are distributing Mac arm64 wheels too; you are welcome ;-):
Read more about SChunk features in our blog entry at: https://www.blosc.org/posts/python-blosc2-improvements
NDArray: an N-Dimensional store
One of the latest and more exciting additions in Python-Blosc2 is the NDArray object. It can write and read n-dimensional datasets in an extremely efficient way thanks to a n-dim 2-level partitioning, allowing to slice and dice arbitrary large and compressed data in a more fine-grained way:
To wet you appetite, here it is how the NDArray object performs on getting slices orthogonal to the different axis of a 4-dim dataset:
We have blogged about this: https://www.blosc.org/posts/blosc2-ndim-intro
We also have a ~2 min explanatory video on why slicing in a pineapple-style (aka double partition) is useful:
Installing
Blosc is now offering Python wheels for the main OS (Win, Mac and Linux) and platforms. You can install binary packages from PyPi using pip:
pip install blosc2Documentation
The documentation is here:
https://blosc.org/python-blosc2/python-blosc2.html
Also, some examples are available on:
Building from sources
python-blosc2 comes with the C-Blosc2 sources with it and can be built in-place:
git clone https://github.com/Blosc/python-blosc2/
cd python-blosc2
git submodule update --init --recursive
python -m pip install -r requirements-build.txt
python setup.py build_ext --inplaceThat’s all. You can proceed with testing section now.
Testing
After compiling, you can quickly check that the package is sane by running the tests:
python -m pip install -r requirements-tests.txt
python -m pytest (add -v for verbose mode)Benchmarking
If curious, you may want to run a small benchmark that compares a plain NumPy array copy against compression through different compressors in your Blosc build:
PYTHONPATH=. python bench/pack_compress.pyLicense
The software is licenses under a 3-Clause BSD license. A copy of the python-blosc2 license can be found in LICENSE.txt.
Mailing list
Discussion about this module is welcome in the Blosc list:
Please follow @Blosc2 to get informed about the latest developments.
Enjoy data!
Release files for blosc2 2.2.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| blosc2-2.2.0.tar.gz | 4.1 MB | Details |
Built distributions (wheels)
Total release size: 76.7 MB
Release files / blosc2-2.2.0.tar.gz
| Download URL | blosc2-2.2.0.tar.gz |
|---|---|
| Size | 4.1 MB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
4cf3893fe85da4bd899ecf7a457fd09b2212fad256182100108de7a92b88aa87
|
|
BLAKE2b-256 checksum How to use checksums |
b6205141526d669b68e7530f417f55771e4bb0bd81fe6e78ccd851d6bae353ba
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.11.2
|
Release files / blosc2-2.2.0-cp311-cp311-win_amd64.whl
| Download URL | blosc2-2.2.0-cp311-cp311-win_amd64.whl |
|---|---|
| Size | 2.2 MB |
| Tags | CPython 3.11 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
5fb7f8075d2ece5cb64ec1a5646dc0cdedfbabb2ea9c03ba1fe88fa58794cef2
|
|
BLAKE2b-256 checksum How to use checksums |
e09edefc6e0d76383d84a5b113007531668feb237303cb1270ea25c5b368f6a8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.11.2
|
Release files / blosc2-2.2.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | blosc2-2.2.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 4.2 MB |
| Tags | CPython 3.11 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
79b40ff98a5c3ce88aabf5b18a76c5ef88b77ce1e6436dc0c0b414b1bc279487
|
|
BLAKE2b-256 checksum How to use checksums |
f6190151f805ac394119cc281f0fb05ac59b1bb971743c5c0c64df48d2823cb9
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.11.2
|
Release files / blosc2-2.2.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
| Download URL | blosc2-2.2.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl |
|---|---|
| Size | 4.1 MB |
| Tags | CPython 3.11 Linux glibc 2.17+ ARM64 |
|
SHA-256 checksum How to use checksums |
52ccee1e3d0337502f6df14cad28a9f747f23b6ff21676ffc77c936714264b87
|
|
BLAKE2b-256 checksum How to use checksums |
654984fa4c9636bf9ee313f6bedda95b1a82f0885254339f915afd2264917e2c
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.11.2
|
Release files / blosc2-2.2.0-cp311-cp311-macosx_11_0_arm64.whl
| Download URL | blosc2-2.2.0-cp311-cp311-macosx_11_0_arm64.whl |
|---|---|
| Size | 3.4 MB |
| Tags | CPython 3.11 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
195d56b75b818c19f196b3e22225b96ed681fa8dda6a740b1966712392bcc445
|
|
BLAKE2b-256 checksum How to use checksums |
2e9fc9c58ad786ce128e4fbaaca7e648a218c186ddcf3967338cfd4c018af854
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.11.2
|
Release files / blosc2-2.2.0-cp311-cp311-macosx_10_9_x86_64.whl
| Download URL | blosc2-2.2.0-cp311-cp311-macosx_10_9_x86_64.whl |
|---|---|
| Size | 4.2 MB |
| Tags | CPython 3.11 macOS 10.9+ x86-64 |
|
SHA-256 checksum How to use checksums |
67b0ab262c5edfada2ea82aeea5958e7ea06eec0cb3a122df3ffdae8dc6de116
|
|
BLAKE2b-256 checksum How to use checksums |
3ce5f80108dbe4b5185114eb64f6fb63f61ce6d03ba9ea90c30a40d7d855c92e
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.11.2
|
Release files / blosc2-2.2.0-cp310-cp310-win_amd64.whl
| Download URL | blosc2-2.2.0-cp310-cp310-win_amd64.whl |
|---|---|
| Size | 2.2 MB |
| Tags | CPython 3.10 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
404d31a4d057b27463b804a9a2975f67579258f849e094b6e92089d8a67c2a62
|
|
BLAKE2b-256 checksum How to use checksums |
6dc0e4e483b6cc98ecdbb2d7f0829c0cb8f187477a371845897654e70d700dae
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.11.2
|
Release files / blosc2-2.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | blosc2-2.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 4.2 MB |
| Tags | CPython 3.10 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
e57ff284b937572f1d01e55ac18bde9d81fdf488c38bfc7cebf9e96454c52e04
|
|
BLAKE2b-256 checksum How to use checksums |
7619e702454e4b1e68987d61948ea4f05944bf153c09fafb48120ca25df0b763
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.11.2
|
Release files / blosc2-2.2.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
| Download URL | blosc2-2.2.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl |
|---|---|
| Size | 4.1 MB |
| Tags | CPython 3.10 Linux glibc 2.17+ ARM64 |
|
SHA-256 checksum How to use checksums |
6758c4f812b7934b071653daa40482461550cf32e06e88098c7fe4b0d11f1247
|
|
BLAKE2b-256 checksum How to use checksums |
c7caefa03f104c44c04c0d5d81aac68e5ef7f832705cfb8d5678556f1e677707
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.11.2
|
Release files / blosc2-2.2.0-cp310-cp310-macosx_11_0_arm64.whl
| Download URL | blosc2-2.2.0-cp310-cp310-macosx_11_0_arm64.whl |
|---|---|
| Size | 3.4 MB |
| Tags | CPython 3.10 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
8b3396334779ebb41dd8722ed165bb3be5da4b6f09420a01bd4d2b15d54bc4cb
|
|
BLAKE2b-256 checksum How to use checksums |
60c77d92041281f808c0f8daa6cbce86edcbe08aeba223874fcf539ea0992f39
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.11.2
|
Release files / blosc2-2.2.0-cp310-cp310-macosx_10_9_x86_64.whl
| Download URL | blosc2-2.2.0-cp310-cp310-macosx_10_9_x86_64.whl |
|---|---|
| Size | 4.3 MB |
| Tags | CPython 3.10 macOS 10.9+ x86-64 |
|
SHA-256 checksum How to use checksums |
317608ae618101899274bd5b4d33c5a8ff3234c9e09da66549e7c67463b41fca
|
|
BLAKE2b-256 checksum How to use checksums |
dffae6766d47fe578d65cc2b528252113e35f1165982e768bec331b729f9f046
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.11.2
|
Release files / blosc2-2.2.0-cp39-cp39-win_amd64.whl
| Download URL | blosc2-2.2.0-cp39-cp39-win_amd64.whl |
|---|---|
| Size | 2.2 MB |
| Tags | CPython 3.9 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
e85a7084f802e04e7fd651975aef8c49ed5d39f58f83ab53b936c99aa7d431f4
|
|
BLAKE2b-256 checksum How to use checksums |
89073fc069a66462f4bd8e46a7c7908a091d777e85b81d4a6a66da1e4649c8f0
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.11.2
|
Release files / blosc2-2.2.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | blosc2-2.2.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 4.2 MB |
| Tags | CPython 3.9 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
f49007f69c7ae7388cde9ca11d988b08e669493665db2b654773d79af58da0ad
|
|
BLAKE2b-256 checksum How to use checksums |
f25ab3f2fef4a4188bace3b26d1e2b2679c6815551223ccd4e222e8f6c53ef88
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.11.2
|
Release files / blosc2-2.2.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
| Download URL | blosc2-2.2.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl |
|---|---|
| Size | 4.1 MB |
| Tags | CPython 3.9 Linux glibc 2.17+ ARM64 |
|
SHA-256 checksum How to use checksums |
2eb94c50b6f507523f03ecb21f9999472471ded04db5200689d06db191c86686
|
|
BLAKE2b-256 checksum How to use checksums |
8e689d21d29e504cd36a1ad9669b91e8b7043441a6b2c489b5a0300db405adc6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.11.2
|
Release files / blosc2-2.2.0-cp39-cp39-macosx_11_0_arm64.whl
| Download URL | blosc2-2.2.0-cp39-cp39-macosx_11_0_arm64.whl |
|---|---|
| Size | 3.4 MB |
| Tags | CPython 3.9 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
9d4a068967e443c3999652f0b946b0c45e617772ccf84fb43fac854919bb72c6
|
|
BLAKE2b-256 checksum How to use checksums |
2bd786e91774234918378adece2794f04b21c2efc216a6fb4c6d32403012dde6
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.11.2
|
Release files / blosc2-2.2.0-cp39-cp39-macosx_10_9_x86_64.whl
| Download URL | blosc2-2.2.0-cp39-cp39-macosx_10_9_x86_64.whl |
|---|---|
| Size | 4.3 MB |
| Tags | CPython 3.9 macOS 10.9+ x86-64 |
|
SHA-256 checksum How to use checksums |
4a52b9f26f00f47b4219192c9192741f068da59db62fd2e7e237434fecdc0e01
|
|
BLAKE2b-256 checksum How to use checksums |
242c845d0e3fb16f69045a8aa6b030226d059064f94cde9b1c0246a22ba89c71
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.11.2
|
Release files / blosc2-2.2.0-cp38-cp38-win_amd64.whl
| Download URL | blosc2-2.2.0-cp38-cp38-win_amd64.whl |
|---|---|
| Size | 2.2 MB |
| Tags | CPython 3.8 Windows x86-64 |
|
SHA-256 checksum How to use checksums |
094c0dd5bb7ad47f2b526d2b43b36768db72a46b3de0a35953d80e40091e51d3
|
|
BLAKE2b-256 checksum How to use checksums |
db0259e5b956c726805669323de707e12c703b892504acd5f16f897ad4c85fcc
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.11.2
|
Release files / blosc2-2.2.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
| Download URL | blosc2-2.2.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl |
|---|---|
| Size | 4.2 MB |
| Tags | CPython 3.8 Linux glibc 2.17+ x86-64 |
|
SHA-256 checksum How to use checksums |
fa7d90390ffb2d4680c9c4c079316e68a8f3177d6938b35da3eae733f0132ef1
|
|
BLAKE2b-256 checksum How to use checksums |
4a8bbf28f7701443ade7029407cc19460ee585e1099bbadf494f453a28670003
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.11.2
|
Release files / blosc2-2.2.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
| Download URL | blosc2-2.2.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl |
|---|---|
| Size | 4.1 MB |
| Tags | CPython 3.8 Linux glibc 2.17+ ARM64 |
|
SHA-256 checksum How to use checksums |
436dad718506a7d9ddf42cdfff7416aef8e2dc97ce65c1e4f221911449dcfa5f
|
|
BLAKE2b-256 checksum How to use checksums |
c8a5b1885514fd7b56aea7c6e6009acfffe9d9f6be8518101e69f0e2c55fa8ed
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.11.2
|
Release files / blosc2-2.2.0-cp38-cp38-macosx_11_0_arm64.whl
| Download URL | blosc2-2.2.0-cp38-cp38-macosx_11_0_arm64.whl |
|---|---|
| Size | 3.4 MB |
| Tags | CPython 3.8 macOS 11.0+ ARM64 |
|
SHA-256 checksum How to use checksums |
510a07cb62ea9dca928b366f346be3849d729fa438c3d500796acb100b90f042
|
|
BLAKE2b-256 checksum How to use checksums |
6dd7031a7b702ccebe14857ba54bd38f9dd8c8e9ec69a2e73e619b8b71e214e8
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.11.2
|
Release files / blosc2-2.2.0-cp38-cp38-macosx_10_9_x86_64.whl
| Download URL | blosc2-2.2.0-cp38-cp38-macosx_10_9_x86_64.whl |
|---|---|
| Size | 4.3 MB |
| Tags | CPython 3.8 macOS 10.9+ x86-64 |
|
SHA-256 checksum How to use checksums |
aac779857e2c6715ba74ec6e72687afc387f9a5dbff90c4ae3c8a5b7d3e59a6c
|
|
BLAKE2b-256 checksum How to use checksums |
9770162f1986a5b79067cc5c553906b67fa890ea21137eb3af32903e9901ad18
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/4.0.1 CPython/3.11.2
|
