Skip to main content

Thin Python bindings to de/compression algorithms in Rust

Project description

pyrus-cramjam

Code Style CI PyPI Anaconda-Server Badge Downloads

API Documentation

Install

pip install --upgrade cramjam  # Requires no Python or system dependencies!

Extremely thin Python bindings to de/compression algorithms in Rust. Allows for using algorithms such as Snappy, without any system dependencies.

This is handy when being used in environments like AWS Lambda, where installing packages like python-snappy becomes difficult because of system level dependencies.


Benchmarks

Some basic benchmarks are available in the benchmarks directory


Available algorithms:

  • Snappy
  • Brotli
  • Lz4
  • Gzip
  • Deflate
  • ZSTD

All available for use as:

>>> import cramjam
>>> import numpy as np
>>> compressed = cramjam.snappy.compress(b"bytes here")
>>> decompressed = cramjam.snappy.decompress(compressed)
>>> decompressed
cramjam.Buffer(len=10)  # an object which implements the buffer protocol
>>> bytes(decompressed)
b"bytes here"
>>> np.frombuffer(decompressed, dtype=np.uint8)
array([ 98, 121, 116, 101, 115,  32, 104, 101, 114, 101], dtype=uint8)

Where the API is cramjam.<compression-variant>.compress/decompress and accepts bytes/bytearray/numpy.array/cramjam.File/cramjam.Buffer objects.

de/compress_into Additionally, all variants support decompress_into and compress_into. Ex.

>>> import numpy as np
>>> from cramjam import snappy, Buffer
>>>
>>> data = np.frombuffer(b'some bytes here', dtype=np.uint8)
>>> data
array([115, 111, 109, 101,  32,  98, 121, 116, 101, 115,  32, 104, 101,
       114, 101], dtype=uint8)
>>>
>>> compressed = Buffer()
>>> snappy.compress_into(data, compressed)
33  # 33 bytes written to compressed buffer
>>>
>>> compressed.tell()  # Where is the buffer position?
33  # goodie!
>>>
>>> compressed.seek(0)  # Go back to the start of the buffer so we can prepare to decompress
>>> decompressed = b'0' * len(data)  # let's write to `bytes` as output
>>> decompressed
b'000000000000000'
>>>
>>> snappy.decompress_into(compressed, decompressed)
15  # 15 bytes written to decompressed
>>> decompressed
b'some bytes here'

Special note!
If you know the length of the de/compress output, you can provide output_len=<<some int>> to any de/compress to get ~1.5-3x performance increase as this allows single buffer allocation; doesn't really apply if you're using cramjam.Buffer or cramjam.File objects.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

cramjam-2.4.0_rc2.tar.gz (1.0 MB view hashes)

Uploaded Source

Built Distributions

cramjam-2.4.0_rc2-pp36-pypy36_pp73-manylinux2010_x86_64.whl (1.5 MB view hashes)

Uploaded PyPy manylinux: glibc 2.12+ x86-64

cramjam-2.4.0_rc2-pp36-pypy36_pp73-macosx_10_7_x86_64.whl (1.4 MB view hashes)

Uploaded PyPy macOS 10.7+ x86-64

cramjam-2.4.0_rc2-cp39-none-win_amd64.whl (937.2 kB view hashes)

Uploaded CPython 3.9 Windows x86-64

cramjam-2.4.0_rc2-cp39-none-win32.whl (859.8 kB view hashes)

Uploaded CPython 3.9 Windows x86

cramjam-2.4.0_rc2-cp39-cp39-manylinux_2_17_s390x.manylinux2014_s390x.whl (1.6 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.17+ s390x

cramjam-2.4.0_rc2-cp39-cp39-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (1.4 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ppc64le

cramjam-2.4.0_rc2-cp39-cp39-manylinux_2_17_ppc64.manylinux2014_ppc64.whl (1.4 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ppc64

cramjam-2.4.0_rc2-cp39-cp39-manylinux_2_17_armv7l.manylinux2014_armv7l.whl (1.4 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ARMv7l

cramjam-2.4.0_rc2-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.3 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.17+ ARM64

cramjam-2.4.0_rc2-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.5 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.5+ x86-64

cramjam-2.4.0_rc2-cp39-cp39-manylinux_2_5_i686.manylinux1_i686.whl (1.5 MB view hashes)

Uploaded CPython 3.9 manylinux: glibc 2.5+ i686

cramjam-2.4.0_rc2-cp39-cp39-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl (2.6 MB view hashes)

Uploaded CPython 3.9 macOS 10.9+ universal2 (ARM64, x86-64) macOS 10.9+ x86-64 macOS 11.0+ ARM64

cramjam-2.4.0_rc2-cp39-cp39-macosx_10_7_x86_64.whl (1.4 MB view hashes)

Uploaded CPython 3.9 macOS 10.7+ x86-64

cramjam-2.4.0_rc2-cp38-none-win_amd64.whl (937.2 kB view hashes)

Uploaded CPython 3.8 Windows x86-64

cramjam-2.4.0_rc2-cp38-none-win32.whl (859.9 kB view hashes)

Uploaded CPython 3.8 Windows x86

cramjam-2.4.0_rc2-cp38-cp38-manylinux_2_17_s390x.manylinux2014_s390x.whl (1.6 MB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.17+ s390x

cramjam-2.4.0_rc2-cp38-cp38-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (1.4 MB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.17+ ppc64le

cramjam-2.4.0_rc2-cp38-cp38-manylinux_2_17_ppc64.manylinux2014_ppc64.whl (1.4 MB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.17+ ppc64

cramjam-2.4.0_rc2-cp38-cp38-manylinux_2_17_armv7l.manylinux2014_armv7l.whl (1.4 MB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.17+ ARMv7l

cramjam-2.4.0_rc2-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.3 MB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.17+ ARM64

cramjam-2.4.0_rc2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.5 MB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.5+ x86-64

cramjam-2.4.0_rc2-cp38-cp38-manylinux_2_5_i686.manylinux1_i686.whl (1.5 MB view hashes)

Uploaded CPython 3.8 manylinux: glibc 2.5+ i686

cramjam-2.4.0_rc2-cp38-cp38-macosx_10_9_x86_64.macosx_11_0_arm64.macosx_10_9_universal2.whl (2.6 MB view hashes)

Uploaded CPython 3.8 macOS 10.9+ universal2 (ARM64, x86-64) macOS 10.9+ x86-64 macOS 11.0+ ARM64

cramjam-2.4.0_rc2-cp38-cp38-macosx_10_7_x86_64.whl (1.4 MB view hashes)

Uploaded CPython 3.8 macOS 10.7+ x86-64

cramjam-2.4.0_rc2-cp37-none-win_amd64.whl (937.2 kB view hashes)

Uploaded CPython 3.7 Windows x86-64

cramjam-2.4.0_rc2-cp37-none-win32.whl (859.8 kB view hashes)

Uploaded CPython 3.7 Windows x86

cramjam-2.4.0_rc2-cp37-cp37m-manylinux_2_17_s390x.manylinux2014_s390x.whl (1.6 MB view hashes)

Uploaded CPython 3.7m manylinux: glibc 2.17+ s390x

cramjam-2.4.0_rc2-cp37-cp37m-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (1.4 MB view hashes)

Uploaded CPython 3.7m manylinux: glibc 2.17+ ppc64le

cramjam-2.4.0_rc2-cp37-cp37m-manylinux_2_17_ppc64.manylinux2014_ppc64.whl (1.4 MB view hashes)

Uploaded CPython 3.7m manylinux: glibc 2.17+ ppc64

cramjam-2.4.0_rc2-cp37-cp37m-manylinux_2_17_armv7l.manylinux2014_armv7l.whl (1.4 MB view hashes)

Uploaded CPython 3.7m manylinux: glibc 2.17+ ARMv7l

cramjam-2.4.0_rc2-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.3 MB view hashes)

Uploaded CPython 3.7m manylinux: glibc 2.17+ ARM64

cramjam-2.4.0_rc2-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.5 MB view hashes)

Uploaded CPython 3.7m manylinux: glibc 2.5+ x86-64

cramjam-2.4.0_rc2-cp37-cp37m-manylinux_2_5_i686.manylinux1_i686.whl (1.5 MB view hashes)

Uploaded CPython 3.7m manylinux: glibc 2.5+ i686

cramjam-2.4.0_rc2-cp37-cp37m-macosx_10_7_x86_64.whl (1.4 MB view hashes)

Uploaded CPython 3.7m macOS 10.7+ x86-64

cramjam-2.4.0_rc2-cp36-none-win_amd64.whl (935.3 kB view hashes)

Uploaded CPython 3.6 Windows x86-64

cramjam-2.4.0_rc2-cp36-none-win32.whl (857.7 kB view hashes)

Uploaded CPython 3.6 Windows x86

cramjam-2.4.0_rc2-cp36-cp36m-manylinux_2_17_s390x.manylinux2014_s390x.whl (1.6 MB view hashes)

Uploaded CPython 3.6m manylinux: glibc 2.17+ s390x

cramjam-2.4.0_rc2-cp36-cp36m-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl (1.4 MB view hashes)

Uploaded CPython 3.6m manylinux: glibc 2.17+ ppc64le

cramjam-2.4.0_rc2-cp36-cp36m-manylinux_2_17_ppc64.manylinux2014_ppc64.whl (1.4 MB view hashes)

Uploaded CPython 3.6m manylinux: glibc 2.17+ ppc64

cramjam-2.4.0_rc2-cp36-cp36m-manylinux_2_17_armv7l.manylinux2014_armv7l.whl (1.3 MB view hashes)

Uploaded CPython 3.6m manylinux: glibc 2.17+ ARMv7l

cramjam-2.4.0_rc2-cp36-cp36m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (1.3 MB view hashes)

Uploaded CPython 3.6m manylinux: glibc 2.17+ ARM64

cramjam-2.4.0_rc2-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (1.5 MB view hashes)

Uploaded CPython 3.6m manylinux: glibc 2.5+ x86-64

cramjam-2.4.0_rc2-cp36-cp36m-manylinux_2_5_i686.manylinux1_i686.whl (1.5 MB view hashes)

Uploaded CPython 3.6m manylinux: glibc 2.5+ i686

cramjam-2.4.0_rc2-cp36-cp36m-macosx_10_7_x86_64.whl (1.4 MB view hashes)

Uploaded CPython 3.6m macOS 10.7+ x86-64

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page