compress-utils 0.4.1

Simple & high-performance compression utilities for Python

compress-utils

compress-utils aims to simplify data compression by offering a unified interface for various algorithms and languages, while maintaining best-in-class performance.

Features

• 4 built-in data compression algorithms
• 3 languages supported
• Standardized API across all algorithms & languages
• Portable & cross-platform (Linux, macOS, Windows)
• Prebuilt binaries available on major package managers or can be built from source
• Native or near-native compression & decompression performance
• Lightweight binary (30 kB with single algorithm, 4 MB with all)

Built-in Compression Algorithms

Algorithm	Description	Benchmarks
brotli	General-purpose with high-to-very-high compression rates	Benchmarks
zlib	General-purpose, widely-used (compatible with gzip)	Benchmarks
zstd	High-speed, high-ratio compression algorithm	Benchmarks
xz/lzma	Very-high compression ratio algorithm	Benchmarks

Supported Languages

Language	Package	Code Examples & Docs
C++	TBD	C++ API
C	TBD	C API
Python	TBD	Python API

Usage

This project aims to bring a unified interface across all algorithms & all languages (within reason). To make this possible across all targeted languages, the compress-utils API is made available in two flavors:

• Object-Oriented (OOP)
• Functional

Both of these APIs are made dead simple. Here's an OOP example in Python:

from compress_utils import compressor

# Create a 'zstd' compressor object
comp = compressor('zstd')

# Compress data
compressed_data = comp.compress(data)

# Compress data with a compression level (1-10)
compressed_data = comp.compress(data, 5)

# Decompress data
decompressed_data = comp.decompress(compressed_data)

Functional usage is similarly simple:

from compress_utils import compress, decompress

# Compress data using `zstd`
compressed_data = compress(data, 'zstd')

# Compress data with a compression level (1-10)
compressed_data = compress(data, 'zstd', 5)

# Decompress data
decompressed_data = decompress(compressed_data, 'zstd')

Language-Specific Examples

You can find language-specific code examples below:

• C++ API Docs >
• C API Docs >
• Python API Docs >

Setup

Install From Package Manager

To be added

Build From Source

1. Install pre-requisites

• CMake
• Conda (if building Python binding)

2. Clone repo

git clone https://github.com/dupontcyborg/compress-utils.git
cd compress-utils

3. Activate Conda environment (if building Python binding)

# If using Conda
conda env create -f environment.yml
conda activate compress-utils

# If using Mamba
mamba env create -f environment.yml
mamba activate compress-utils

4. Run build script

For Linux/macOS:

build.sh

For Windows:

powershell.exe -file build.ps1

The built library/libraries will be in dist/<language>

A number of configuration parameters are available for build.sh:

• --clean - performs a clean rebuild of compress-utils
• --algorithms= - set which algorithms to include in the build, if not all (e.g., build.sh --algorithms=brotli,zlib,zstd)
• --languages= - set which language bindings to build, if not all (e.g., build.sh --languages=python,js)
• --release - build release version (higher optimization level)
• --skip-tests - skip building & running unit tests

Benchmarks

To be added

License

This project is distributed under the MIT License. Read more >

Third-Party Code

This project utilizes several open-source compression algorithms. Read more >