NumPy Cache – Fast LZ4-Based Caching for NumPy Arrays

High‑performance, lightweight disk cache for NumPy arrays with LZ4 compression – now with configurable compression speed.

📌 The Problem

When dealing with large NumPy arrays, developers face a classic trade‑off:

There is a clear gap: no lightweight, specialised solution combines np.save() speed with good compression – until now.

🚀 Features

• ✅ Blazing fast – 20 × faster than np.savez_compressed()
• ✅ Good compression – 2 × smaller than np.save()
• ✅ Pure C extension – minimal overhead, maximum performance
• ✅ Configurable speed – acceleration parameter (1–16) lets you trade compression ratio for speed
• ✅ NumPy integration – works with all numeric dtypes (int, uint, float, bool)
• ✅ Multi‑dimensional – supports up to 8 dimensions
• ✅ Contiguous arrays – automatically handles non‑contiguous slices
• ✅ Empty arrays – properly saves and loads zero‑size arrays
• ✅ Lightweight – no external dependencies beyond NumPy and LZ4
• ✅ Apache 2.0 License – free for commercial and personal use

📦 Installation

System Dependencies

First, install the LZ4 library:

# Ubuntu / Debian
sudo apt-get install liblz4-dev

# macOS (Homebrew)
brew install lz4

# Fedora / RHEL
sudo dnf install lz4-devel

# Arch Linux
sudo pacman -S lz4

Supported Python versions: 3.12, 3.13, 3.14, 3.15

Install from PyPI

pip install numpy-cache

Install from source

git clone https://github.com/macht1212/numpy-cache.git
cd numpy-cache
poetry install

🧪 Usage

import numpy as np
from numpy_cache import save, load

# Create a large array
arr = np.random.randn(5000, 5000).astype(np.float32)

# Save with LZ4 (default acceleration = 4)
save(arr, 'my_array.npc')

# Control compression speed vs. ratio
# acceleration=1 → best compression, slower
# acceleration=16 → fastest, slightly worse compression
save(arr, 'my_array_fast.npc', acceleration=16)

# Load back
loaded = load('my_array.npc')

# Verify
np.testing.assert_array_equal(arr, loaded)

Acceleration Parameter

• 1–4: Better compression ratio, slower.
• 5–10: Balanced default (4 is recommended).
• 11–16: Maximum speed, slightly larger files.

Supported Dtypes

All NumPy numeric types are supported:

• float32, float64
• int8, int16, int32, int64
• uint8, uint16, uint32, uint64
• bool_

Multi‑dimensional Arrays

arr_3d = np.random.randn(100, 100, 100)
save(arr_3d, '3d_array.npc')

Non‑contiguous Slices

arr = np.random.randn(1000, 1000)
slice_arr = arr[::2, ::2]  # Not contiguous
save(slice_arr, 'slice.npc')  # Handles automatically

📊 Benchmarks

Test system:

• Ubuntu 24.04.4 LTS, 12th Gen Intel i5-1235U (12 cores), 16 GB RAM, NVMe SSD
• Python 3.12, NumPy 2.5.2, LZ4 1.9.4

All arrays are float32. Sizes:

• shape0: 100×100 = 10 000 elements ≈ 0.04 MB
• shape1: 500×500 = 250 000 elements ≈ 1 MB

Write Performance (time in μs)

Read Performance (time in μs)

File Size Comparison (1 MB array)

• np.save / np.savez: ~1.0 MB
• np.savez_compressed: ~0.5 MB (varies)
• numpy_cache: ~0.4 MB (depends on acceleration)

Key Takeaways

numpy_cache is 24–40× faster than np.savez_compressed for writes.

For reads, it is 5–12× faster than np.savez_compressed.

Compression ratio is better than np.savez and usually close to np.savez_compressed.

The acceleration parameter lets you fine‑tune the speed/ratio trade‑off.

🛠️ How It Works

Architecture

1. Pure C Extension – compiled into a Python module for maximum performance.
2. LZ4 Compression – uses LZ4_compress_fast() with configurable acceleration.
3. Custom Binary Format – packed header (96 bytes) + compressed payload.
4. Direct NumPy Integration – zero‑copy access to array data where possible.

File Format

The header is packed (no padding) to ensure portability:

#pragma pack(push, 1)
typedef struct {
    uint64_t uncompressed_size;
    uint64_t compressed_size;
    uint64_t shape[MAX_DIMS];   // up to 8 dimensions
    uint32_t magic;             // 0x4C5A4E43 ("LZNC")
    uint32_t version;           // 1
    uint32_t ndim;
    uint32_t dtype;             // NumPy type ID
} CacheHeader;
#pragma pack(pop)

• Magic identifies the file format.
• Version allows future upgrades.
• The header is followed immediately by the LZ4‑compressed data.

Project Structure

numpy_cache/
├── csrc/
│   └── cache_module.c      # C extension
├── src/
│   └── numpy_cache/
│       ├── __init__.py     # Python wrapper
│       └── _cache.so       # Compiled extension
├── tests/
│   ├── test_cache.py       # Unit tests
│   └── test_benchmarks.py  # Performance benchmarks
├── setup.py                # Setuptools configuration
├── pyproject.toml          # Poetry configuration
├── CHANGELOG.md
└── README.md

Build from Source

# Install development dependencies
poetry install

# Build the C extension
poetry run python setup.py build_ext --inplace

# Run tests
poetry run pytest

# Run benchmarks
poetry run pytest tests/test_benchmarks.py --benchmark-only

Run Benchmarks Separately

# Write benchmarks
poetry run pytest tests/test_benchmarks.py -k "write" --benchmark-only

# Read benchmarks
poetry run pytest tests/test_benchmarks.py -k "read" --benchmark-only

🗺️ Roadmap (Future Improvements)

• Multi‑threaded compression – parallelise LZ4 for even faster saving of huge arrays.
• Asynchronous I/O – background saving without blocking the main thread.
• Progress bar – visual feedback for very large arrays (via tqdm integration).
• Windows support – ensure compatibility with MSVC and the Windows API.
• Zstd backend – optional support for Zstandard compression (better ratio).
• Memory mapping – load arrays directly from disk without full decompression (for streaming).

📄 License

This project is licensed under the Apache License, Version 2.0 – see the LICENSE file for details.

🙏 Acknowledgments

• LZ4 – extremely fast compression library.
• NumPy – fundamental array computing.
• Python – the language that makes it all possible.

Happy caching! 🚀