Skip to main content
Pre-release

This release is a pre-release and may not be stable for production use.

bslz4_to_sparse

Decompress Dectris Eiger bitshuffle-LZ4 data directly to sparse index/value arrays.

Documentation, with runnable examples and speed on real data: https://jonwright.github.io/bslz4_to_sparse/

This repository brings together upstream code from bitshuffle [1], the DECTRIS bitshuffle SIMD kernels [2], LZ4 [3] and Zstandard [4] to process detector data as a stream (in cache), rather than writing whole images to RAM and reading them back again. Versions before 0.0.16 were mostly written by @jonwright; later versions include a large amount of LLM-generated code (DeepSeek and Claude).

It is used in ImageD11 [5] to convert images directly to sparse format, and it can also do radial integrations as sparse matrix products (for example with the integration matrices of pyFAI [6]). The C/C++ core is a Python extension through c2py23 [7]. Each decoder object has a pipeline: one value per processing step (decode, mask, untranspose, collect, and for matrix objects route and dot), chosen automatically for the CPU, data and matrix unless given; bslz4_to_sparse.describe() lists the values and which ones this machine can run.

After git clone:

git submodule update --init
python3 -m pip install .

To run it at the ESRF:

cd test
python3 test_vs_hdf5plugin.py
python3 bench1.py

To test a local build without installing:

python3 setup.py build --build-lib=lib
cd test
BSLZ4_TO_SPARSE_PATH=$(pwd)/../lib python3 -m pytest -v
BSLZ4_TO_SPARSE_PATH=$(pwd)/../lib python3 bench1.py

Every script in test/ prepends BSLZ4_TO_SPARSE_PATH to sys.path, so it wins over any installed bslz4_to_sparse and works where PYTHONPATH is ignored (ESRF jupyter-slurm).

The extension is tagged by platform only (_bslz4_to_sparse.c2py23-linux_ppc64le.so); c2py23 resolves the CPython API at runtime, so one binary serves any Python version. Run the build once per machine: a single lib/ on a shared filesystem holds all of them. C2PY_TRACE=1 reports which file the loader picks.

Threads and free-threaded Python (3.14t, 3.15t): the decode entry points release the GIL, and the extension declares itself free-threading safe, so importing it does not re-enable the GIL. Nothing is locked; the rule is one chunk2sparse / chunk2sparseCSC object per thread (for the second, give the objects of a thread pool one plan(mask, csc): they share its matrix instead of each building and holding a copy). Each object reuses its own workspace and output arrays, so the arrays a call returns are overwritten by that object's next call, and a call's inputs (chunks, mmap base, mask) must not be changed or closed by another thread while it runs. Breaking these rules gives wrong results or a crash, not an error. _testing.counters() (test instrumentation) is only exact while one decode runs at a time.

Collect values the machine cannot run (vsx on x86_64, neon on x86_64/POWER, avx512cs/avx2cs/sse2 on POWER/ARM) are skipped by test_simd_collect_matches_scalar.

The licences of the bundled upstream code are in licenses/.

Acknowledgements

This package is mostly a regrouping of other people's work, put together so that detector frames can be decoded and reduced in one pass. The bitshuffle format and its reference implementation come from K. Masui and co-authors [1]; the fast SIMD bit-transpose kernels from K. Conley at DECTRIS [2]; the LZ4 and Zstandard decompressors from Y. Collet and the Zstandard contributors [3], [4]. The integration matrices are pyFAI's [6], and the sparse output serves ImageD11 [5]. The optional GPU decoder uses an edited extract of the LZ4 device kernels of NVIDIA's nvcomp [8]. The portable Python wrappers come from c2py23 [7]: they let one compiled module load on every CPython from 2.7 to 3.15, free-threaded builds included, without compiling against any Python headers. Our thanks to all of these projects; please cite them as well as this one.

References

[1] bitshuffle, K. Masui et al., https://github.com/kiyo-masui/bitshuffle; K. Masui et al., "A compression scheme for radio data in high performance computing", Astronomy and Computing 12, 181-190 (2015), https://doi.org/10.1016/j.ascom.2015.07.002

[2] bitshuffle SIMD bit-transpose kernels ("kcb"), K. Conley (DECTRIS), https://github.com/kalcutter/bitshuffle

[3] LZ4, Y. Collet, https://github.com/lz4/lz4

[4] Zstandard, Y. Collet and Meta Platforms, https://github.com/facebook/zstd; Y. Collet and M. Kucherawy (ed.), "Zstandard Compression and the 'application/zstd' Media Type", RFC 8878 (2021), https://doi.org/10.17487/RFC8878

[5] ImageD11, https://github.com/FABLE-3DXRD/ImageD11

[6] pyFAI, https://github.com/silx-kit/pyFAI; G. Ashiotis, A. Deschildre, Z. Nawaz, J. P. Wright, D. Karkoulis, F. E. Picca and J. Kieffer, "The fast azimuthal integration Python library: pyFAI", J. Appl. Cryst. 48, 510-519 (2015), https://doi.org/10.1107/S1600576715004306

[7] c2py23, https://github.com/jonwright/c2py23

[8] nvcomp, NVIDIA, https://github.com/NVIDIA/nvcomp (BSD 3-Clause; src/cuda/nvcomp_lz4.cuh)

Metadata

Release files for bslz4-to-sparse 0.0.22a1

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for bslz4-to-sparse 0.0.22a1
File Size Uploaded Provenance Actions
bslz4_to_sparse-0.0.22a1.tar.gz 1.1 MB With provenance Details for bslz4_to_sparse-0.0.22a1.tar.gz

Built distributions (wheels)

Table of built distributions (wheels) for bslz4-to-sparse 0.0.22a1
File Actions
bslz4_to_sparse-0.0.22a1-py2.py3-none-win_amd64.whl Python 2, Python 3 none Windows x86-64 Details for bslz4_to_sparse-0.0.22a1-py2.py3-none-win_amd64.whl
bslz4_to_sparse-0.0.22a1-py2.py3-none-manylinux_2_17_x86_64.whl Python 2, Python 3 none Linux glibc 2.17+ x86-64 Details for bslz4_to_sparse-0.0.22a1-py2.py3-none-manylinux_2_17_x86_64.whl
bslz4_to_sparse-0.0.22a1-py2.py3-none-manylinux_2_17_ppc64le.whl Python 3, Python 2 none Linux glibc 2.17+ PowerPC 64-le Details for bslz4_to_sparse-0.0.22a1-py2.py3-none-manylinux_2_17_ppc64le.whl
bslz4_to_sparse-0.0.22a1-py2.py3-none-manylinux_2_17_aarch64.whl Python 2, Python 3 none Linux glibc 2.17+ ARM64 Details for bslz4_to_sparse-0.0.22a1-py2.py3-none-manylinux_2_17_aarch64.whl
bslz4_to_sparse-0.0.22a1-py2.py3-none-macosx_11_0_arm64.whl Python 2, Python 3 none macOS 11.0+ ARM64 Details for bslz4_to_sparse-0.0.22a1-py2.py3-none-macosx_11_0_arm64.whl

Total release size: 2.8 MB

Release files / bslz4_to_sparse-0.0.22a1.tar.gz

Download URL bslz4_to_sparse-0.0.22a1.tar.gz
Size 1.1 MB
Tags Source
SHA-256 checksum
How to use checksums
e2a83ffb1c95078dda803a9504ab6a2dd539d337eef61a212530672aec4ee2e9
BLAKE2b-256 checksum
How to use checksums
a64bd545c91dfb3f4f30cacec9322079a544bd83ef4ac9eb891d75a4d8972936
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 16, 2026.

Transparency log

Release files / bslz4_to_sparse-0.0.22a1-py2.py3-none-win_amd64.whl

Download URL bslz4_to_sparse-0.0.22a1-py2.py3-none-win_amd64.whl
Size 471.6 kB
Tags Python 2 Python 3 Windows x86-64
SHA-256 checksum
How to use checksums
3d70dbfc5e63538bdb76597dc3a2eeb2da070c24658599e740287b0bf0bdd8c2
BLAKE2b-256 checksum
How to use checksums
e2684961a97c1552e8adee3d9700866dee9a8eb8856cc3d508b6cb311ba453bb
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 16, 2026.

Transparency log

Release files / bslz4_to_sparse-0.0.22a1-py2.py3-none-manylinux_2_17_x86_64.whl

Download URL bslz4_to_sparse-0.0.22a1-py2.py3-none-manylinux_2_17_x86_64.whl
Size 333.7 kB
Tags Linux glibc 2.17+ x86-64 Python 2 Python 3
SHA-256 checksum
How to use checksums
b14271e3946ef54f22ade910409415b1134cca44987267f2f340e7756fbfe60d
BLAKE2b-256 checksum
How to use checksums
5820bd92f6af339f53781e90e71d1916b3b10858b88ff661fa8c716b37ba0e14
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 16, 2026.

Transparency log

Release files / bslz4_to_sparse-0.0.22a1-py2.py3-none-manylinux_2_17_ppc64le.whl

Download URL bslz4_to_sparse-0.0.22a1-py2.py3-none-manylinux_2_17_ppc64le.whl
Size 343.2 kB
Tags Linux glibc 2.17+ PowerPC 64-le Python 2 Python 3
SHA-256 checksum
How to use checksums
549e564197d98dbdc939d12d3ea37298160798fc5cca56e23803de8a236245ce
BLAKE2b-256 checksum
How to use checksums
05d5cfd46bdca46f198a9d954336937e9e12fda3ef626606ed6c8e6bb99c9200
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 16, 2026.

Transparency log

Release files / bslz4_to_sparse-0.0.22a1-py2.py3-none-manylinux_2_17_aarch64.whl

Download URL bslz4_to_sparse-0.0.22a1-py2.py3-none-manylinux_2_17_aarch64.whl
Size 278.4 kB
Tags Linux glibc 2.17+ ARM64 Python 2 Python 3
SHA-256 checksum
How to use checksums
eb5809e5ead03f19bdb5e9328f80b48d1ddf3563d654af632a3dd995555c2186
BLAKE2b-256 checksum
How to use checksums
5ad13c7bad040cc4bbfc4c6cfb0f108eb48c9f1d8eb2b06d80efff1905e1199a
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 16, 2026.

Transparency log

Release files / bslz4_to_sparse-0.0.22a1-py2.py3-none-macosx_11_0_arm64.whl

Download URL bslz4_to_sparse-0.0.22a1-py2.py3-none-macosx_11_0_arm64.whl
Size 270.7 kB
Tags Python 2 Python 3 macOS 11.0+ ARM64
SHA-256 checksum
How to use checksums
587a501ff13626b5460270b3e8437599719b1af28d4839193d8b21764cf86592
BLAKE2b-256 checksum
How to use checksums
6b01a63b5f10c4cfe10b147708967d15179bc11ef3b2577ffced7b6dc168bb9b
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing?
Yes
Uploaded via twine/7.0.0 CPython/3.13.14

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 16, 2026.

Transparency log

Security for bslz4-to-sparse 0.0.22a1

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

Full provenance

All 6 files in this release have been uploaded with provenance metadata.

Select from 6 files

Attestation for bslz4_to_sparse-0.0.22a1.tar.gz

PyPI Publish Attestation

PyPI verified that this artifact, at this checksum, originated from the publisher listed below.

Signed by GitHub Actions, verified by PyPI on Oct 16, 2026.

Transparency log

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page