spx-codec 1.0.0

SPX: High Throughput Lossless Image Compression Engine

SPX (Space Express): High Throughput Lossless Image Compression Engine

SPX (Space Express) is a lossless image compression engine using a Hybrid Python/Rust Architecture, featuring Entropy Sharding and Rayon-accelerated 4-way Interleaved rANS to achieve balance between compression ratio and speed. This project exhibits comparable compression ratios through contextual sharding and native computational kernels, bridging Python's flexibility with Rust's native performance.

繁體中文版本 (Traditional Chinese)

---

Table of Contents

1. v1.0.0 Performance Snapshot
2. Technical Analysis
3. Comparison with Existing Formats
4. System Requirements & Installation
5. Quick Start
6. Technical Architecture & Execution Flow
7. Performance Benchmarking
8. Comparative Benchmarks
9. Limitations & Roadmap
10. Dataset Sources
11. Project Background
12. Acknowledgments

---

SPX is built around a clear principle:

Maximize compression efficiency per unit of compute.

Instead of pursuing absolute compression ratio at any cost, SPX focuses on:

• Predictable Performance: Constant-time complexity relative to input resolution.
• Single-Pass Encoding: Non-iterative execution without brute-force search.
• Minimal Modeling Complexity: Stateless single-model pipeline.
• High Throughput: ILP-optimized native computational kernels.

Trade-off Philosophy

Dimension	SPX Approach
Compression	Competitive (aligned with modern lossless standards)
Speed	$O(N)$ Complexity (non-iterative)
Complexity	Minimal (stateless pipeline)
Determinism	Absolute
Multi-pass	No

Key Characteristics

• Single-pass encoding: Eliminates iterative refinement loops.
• Deterministic pipeline: Constant execution path without heuristic search.
• Reduced model complexity: Single-model context mapping without switching.
• Throughput-centric design: Optimized for maximum pixel-per-cycle throughput.
• Native Rust backend: Zero-cost abstraction with predictable runtime performance.
• Extensible Architecture: Modular configuration of shard boundaries and rANS probability templates to accommodate specialized data distributions.

---

v1.0.0 Performance Snapshot

The following data characterizes the throughput and compression efficiency across standard datasets.

Dataset	Type	SPX BPP	Savings (vs PNG)	Savings (vs PNM)	SPX Enc Speed	WebP (M6) Speed	JXL (E7) Speed
Kodak	RGB	9.79	-24.64 %	-59.19 %	41.24 MB/s	0.33 MB/s	5.65 MB/s
CLIC '25	RGB	8.06	-28.32 %	-66.43 %	69.10 MB/s	1.18 MB/s	5.02 MB/s
CLIC '21	RGB	8.46	-28.03 %	-64.74 %	72.22 MB/s	0.97 MB/s	5.62 MB/s
DIV2K Val	2K	9.22	-27.32 %	-61.59 %	80.76 MB/s	1.28 MB/s	5.83 MB/s
DIV2K Train	2K	9.35	-26.22 %	-61.04 %	66.97 MB/s	1.38 MB/s	6.15 MB/s
Tecnick	RGB	5.18	-25.90 %	-78.42 %	25.12 MB/s	0.68 MB/s	4.75 MB/s
Tecnick	Gray	1.68	-27.63 %	-79.01 %	14.20 MB/s	0.29 MB/s	5.53 MB/s
Standard (ICI)	RGB	10.51	n/a	-56.19 %	150.81 MB/s	2.16 MB/s	10.84 MB/s
Standard (ICI)	Gray	3.44	n/a	-56.99 %	53.06 MB/s	0.58 MB/s	12.16 MB/s

[!NOTE] Hardware Benchmark Environment:

• CPU: AMD Ryzen 5 3500X (6-Core, 3.60 GHz)
• RAM: 32.0 GB
• OS: Windows 11 (64-bit, x64)

Technical Comparison

• Encoding Performance: SPX v1.0.0 exhibits a 103x parallel encoding lead over WebP (m=6) in RGB and a 63x lead in Grayscale (ICI). It remains 6x–8x faster than JXL (Effort 7) across industrial datasets.ted hardware across datasets.
• Loseless Assurance: Bit-perfect reconstruction across all 1,500+ test images (MSE = 0.00000000).
• Core Efficiency: Rust-native backend utilizes Rayon for internal data parallelism and 4-way interleaved rANS for instruction-level parallelism (ILP).
• Hybrid Performance: Critical hot-paths (RCT, MED, Sharding, rANS) are implemented in Rust, while orchestration remains in Python.

Full comparative analysis vs. standard formats is available in Comparative Benchmarks.

---

v1.0.0 Technical Analysis (Hybrid Rust Architecture)

• Core Workflow: Unified pipeline featuring Foundational Protocol $\rightarrow$ Rust Prediction Kernels $\rightarrow$ Rust Spatial Transforms $\rightarrow$ Rust Stateless Sharding.
• Predictor Hub (Pillar 2): Decoupled hub in predictor.py (orchestration) and rans_core.rs (execution) featuring Branchless Edge-Tuned MED for increased execution efficiency.
• Context-Aware Path Architecture:
  • RGB Path (Pillar 3/4): Rust-native G-sub RCT transform coupled with a stateless sharding matrix.
  • Grayscale Fast-Path: Specialized monochrome bypass utilizing serialized Green-channel isolation.
• Stateless Sharding Hub (Pillar 4): Unified profile-driven context ID derivation using the ShardProfile configuration-as-data model, executed via Rust-native "Gather" kernels.
• BICC (Bias Cancellation): Context-driven PDF centering applied to residuals to reduce dispersion.
• Bitplane rANS: Hierarchical entropy modeling using a 2,688-way context model (42 Shards x 64 Spatial Patterns), fully implemented in Rust.
• 4-Way Interleaved rANS Core: Vectorized native entropy engine utilizing instruction-level parallelism (ILP).

---

2. Comparison with Existing Formats

• Compression: ~25-30% reduction compared to standard PNG; comparable with WebP (m6) on high-resolution photography.
• Efficiency: Stateless sharding provides a stable performance profile for both high-frequency noise and low-entropy gradients.
• Decoding: Rust-native decompression throughput (~20–130 MB/s depending on image complexity). Scalable via multi-core batching.

---

3. System Requirements & Installation

• Python Version: 3.10+ (3.11+ recommended)
• Python Packages:
  • numpy>=1.22.0, zstandard>=0.19.0, Pillow>=9.0.0, pytest>=7.0.0
• Native Extension:
  • spx_rans (Rust-native backend)
• Development Dependencies:
  • maturin>=1.0.0 (for bridging Rust and Python)
  • Rust Toolchain: cargo, rustc (required to build the native extension from source)
• System Core (Linux):
  • Requires zlib and libpng headers for Pillow and Zstd I/O (sudo apt install build-essential zlib1g-dev libpng-dev).
• Windows: Self-contained (requires Visual Studio C++ Build Tools if building from source).

[!TIP] Native Acceleration: SPX v1.0.0 utilizes a pre-compiled Rust backend. Unlike previous versions, there is zero JIT latency during the first run. Multithreading: Parallelism is handled internally by the Rust backend using the Rayon library.

Installation:

# 1. Install Python dependencies
pip install numpy>=1.22.0 zstandard>=0.19.0 Pillow>=9.0.0 pytest>=7.0.0

# 2. Build/Install native extension
cd native && maturin develop --release

---

4. Quick Start

4.1 Command Line Interface (CLI)

# Compress
python main.py compress input.png --optimize

# Decompress
python main.py decompress input.spx --output restored.png

# Benchmark (SPX only)
python main.py benchmark ./path/to/images -n 20 -w 8

# Benchmark (Compare SPX vs WebP vs JXL)
python main.py benchmark ./path/to/images --codec bench -n 20

4.2 Python API

from core import compress_spx, decompress_spx

# 1. Compress Image (RGB/RGBA)
result = compress_spx("input.png", "output.spx", use_bitplane=False)
print(f"Ratio: {result.ratio:.2%} | Time: {result.enc_time:.2f}s")

# 2. Decompress Image
with open("output.spx", "rb") as f: payload = f.read()
rgb_arr, dec_time = decompress_spx(payload, "reconstructed.png")
print(f"Dec Time: {dec_time:.2f}s")

4.3 Windows Batch Utility (test.bat)

For Windows users, a convenient batch wrapper is provided for benchmarking:

# Run comparative benchmark (SPX vs WebP vs JXL)
.\test bench ./data/local_test_folder
.\test webp ./data/local_test_folder
.\test jxl ./data/local_test_folder

# Run solo tests for specific codecs
.\test spx ./data/local_test_folder

# Pass additional arguments (e.g., limit to 10 images)
.\test spx ./my_images -n 10

[!NOTE] The data/ directory and benchmark datasets are not included in this repository. To run benchmarks, you can manually create a data/ folder and populate it with your own images (e.g., Kodak, DIV2K) or simply reference the path to your image folder. The utility supports both raw absolute/relative paths and pre-defined aliases (e.g., clic, kodak, trgb) which are managed in core/test_suite.py.

4.4 Project Structure

.
├── core/                   # SPX 4-Pillar Core Engine (Python Orchestration)
│   ├── codec.py            # Bitstream orchestration & serialization
│   ├── sharding.py         # Pillar 4: Shard profiles & Stateless Hub
│   ├── rans.py             # Pillar 4: 4-way interleaved rANS core
│   ├── predictor.py        # Pillar 2: Branchless MED kernels
│   ├── transform.py        # Pillar 3: G-sub RCT & Spatial ops
│   ├── common.py           # Pillar 1: Protocol constants & Flags
│   └── env.py              # Environment & Dependency validator
├── technical/              # Deep-dive algorithmic specifications
├── data/                   # [User-provided] Directory for benchmark datasets (not in repo)
├── native/                 # [Experimental] Rust-accelerated backend
├── test.bat                # Windows benchmark utility
└── main.py                 # CLI entry point

---

5. Technical Architecture & Execution Flow

SPX follows a strictly defined 4-Pillar Architecture to transform raw pixels into a bit-perfect compressed stream:

5.1 The 4 Pillars of SPX

1. Pillar 1: Spatial Transforms (RCT): Handles color decorrelation via the Green-Subtract RCT (transform.py).
2. Pillar 2: Spatial Prediction (MED): Performs spatial decorrelation via Branchless Edge-Tuned MED (predictor.py).
3. Pillar 3: Stateless Sharding: Maps residuals into statistical contexts (shards) for prioritized coding (sharding.py).
4. Pillar 4: Entropy Coding (rANS): Executes statistical compression via the 4-way interleaved rANS engine (rans.py).

These pillars are governed by the Foundational Protocol (common.py), which defines the bitstream schema and coding thresholds.

graph TD
    A[Input: 8-bit RGB/RGBA] --> B{Protocol Gate: Flag Check}
    B -->|Grayscale| C1[Rust: Fused Gray Pass]
    B -->|Color| C2[Rust: Pillar 1 & 2 Fused Kernel]
    
    C1 & C2 --> D[Rust: Pillar 3 Sharding Hub]
    
    D --> E[BICC Bias Cancellation]
    E --> F[Rust: Pillar 4 rANS Engine]
    F --> G[v1.0.0 SPX Bitstream Output]
    
    subgraph "Rust Extension (spx_rans)"
        C1
        C2
        D
        F
    end

5.2 Deep-Dive Technical Series

For detailed algorithmic specifications, refer to the following documentation in the technical/ directory:

• 01. PREDICTOR.md: Detailed logic of the Spatial Dispatcher and MED variations.
• 02. SHARD_TEMPLATE.md: Specification of the Universal-42 matrix and context derivation.
• 03. RANS_MODE.md: Architecture of the 4-way Interleaved rANS core and probability modeling.
• 04. DATASET_FINGERPRINT.md: Statistical performance profiling across industrial datasets.

5.3 Dual-Path Strategy

SPX implements a Context-Aware Bypass logic to handle different image types with optimal efficiency:

• RGB Route: Utilizing the G-sub RCT, it extracts a Green foundation (Lead) followed by RD/BD residuals (Lag). It uses a staggered processing window to maintain context consistency across channels.
• Grayscale Route: If R=G=B is detected, the engine activates a specialized monochrome bypass, reducing computational overhead by ~65%. This path utilizes Bitplane rANS, which decomposes the 8-bit signal into hierarchical layers to increase redundancy extraction efficiency.

5.4 Stateless Sharding & Profile-Driven Hub

The backbone of SPX is the Stateless Sharding Hub, mapping pixels into 42+ contexts based on V-Tier (gradient strength), Intensity, and Trend. This configuration-as-data model allows for seamless profile switching without kernel recompilation.

For entropy coding, the engine utilizes a 30-Mode Template Matrix:

• 10 Base Centroids: Data-driven probability shapes derived from real-world image shards (Hybrid Elite V10).
• 3 Sigma Scales: Each centroid is scaled at 0.5, 1.0, and 1.5 to adapt to different noise levels.
• Zero-Overhead: These 30 empirical modes are hardcoded in the decoder, allowing optimal PDF matching without the "Header Tax" of custom frequency tables.

5.5 Bitplane rANS & Entropy Core

For high-density images, SPX employs Shard-Conditioned Bitplane rANS. Instead of treating the residual as a single 256-symbol alphabet, it decomposes the signal into 2-bit layers. Each layer uses a 2,688-way context model ($42 \text{ Shards} \times 64 \text{ Spatial Patterns}$), allowing the rANS core to isolate structural predictable bits from stochastic noise bits.

The Interleaved rANS engine increases throughput by managing 4 independent state variables in a single loop, increasing CPU execution port utilization via ILP.

5.6 Memory Profile & Scalability

SPX is designed for low-latency processing with a predictable memory footprint.

• Peak RAM (1080p RGB): ~85 MB
• Peak RAM (4K RGB): ~320 MB
• Peak RAM (8K RGB): ~1.2 GB Note: Memory usage scales linearly with pixel count. Peak values include native Rust buffers and Python object overhead.

5.7 Cache Residency & Hot-Path Efficiency

SPX is optimized for L1/L2 cache residency to avoid memory stalls:

• L1 (Hot Path): rANS Models, ZigZag LUTs, Branchless Predictors.
• L2 (Context Hub): Stateless Spatial LUT (256 KB), Shard Histograms.
• L3 (Bulk Data): Image Channel Buffers and Bitstream Payloads. Full technical audit available in technical/CACHE.md.

5.8 Extensibility: Custom Shard Profiles

Pillar 4 (Stateless Sharding) allows adding new segmentation strategies without logic changes:

1. Define Boundaries: Create new V_BOUND and INTENSITY_SEG numpy arrays in sharding.py.
2. Map Shards: Implement a build_shard_map_custom() function to assign Context IDs.
3. Precompute LUTs: Call precompute_luts() to generate the O(1) dispatch tables.
4. Register Profile: Instantiate a new ShardProfile and update the PROFILE_RGB alias.

5.9 Developer Setup & Debugging

• Regression Testing: Run pytest to verify bit-perfect parity and logic integrity.
• Log Verbosity: Set SPX_LOG_LEVEL=DEBUG for detailed pipeline tracing.
• Diagnostic Dumps: Use SPX_DUMP_SHARDS=1 to audit raw residual distributions.
• Thread Safety: Core modules utilize threading.local() for scratch-buffer management; ensure clear_spx_workspaces() is called in long-running server workers.

---

6. Performance Benchmarking (v8.x Unified Hub)

The current engine is benchmarked using the SPX Unified Hub, providing comparative analysis against WebP (Method 6) and JPEG-XL (Effort 7).

6.1 Comprehensive Metrics

Detailed performance data, including compression savings, throughput (MB/s), and competitive win rates, are maintained in technical/BENCHMARK.md.

6.2 Benchmark Baseline Versions

To ensure reproducibility, the competitive baselines are locked to the following versions:

• WebP (Method 6): cwebp v1.3.2 (libwebp v1.3.2).
• JPEG-XL (Effort 7): cjxl v0.8.2 (libjxl v0.8.2).

6.3 Usage

# Use the main entry point
python main.py benchmark C:\datasets\my_images -n 50

# Windows Shortcut (Batch)
.\test bench ./local_folder -n 50

6.4 Technical Control Arguments

Common arguments supported by the benchmarking suite:

Argument	Full Name	Description	Example
-n	--num_tests	Limits the number of processed files.	-n 50
--offset	--offset	Skips the first N images in the set.	--offset 100
-w	--workers	Manually sets the number of CPU cores.	-w 8
--codec	--codec	Selects codec: spx, webp, jxl, bench.	--codec bench
--reclassify	--reclassify	Categorize images into Easy/Hard/Hell folders.	--reclassify
--bitplane	--bitplane	Force the Bitplane engine for the benchmark.	--bitplane
--build	--build	Assemble dataset: PATH E H HELL.	--build my_set 10 10 5

Advanced CLI Examples

# 1. Analyze a specific slice of a large dataset
python main.py benchmark C:\data -n 100 --offset 500

# 2. Extract specific difficulty levels from a results run
# This copies processed files into ./data/DIV2K_Easy, etc.
python main.py benchmark ./my_images --reclassify

# 3. Create a synthetic dataset (10 Easy, 10 Hard, 5 Hell images)
# Resulting images are saved to './data/balanced_set'
python main.py benchmark --build balanced_set 10 10 5

---

7. Comparative Benchmarks (CLIC / DIV2K / TECNICK / KODAK)

Detailed performance metrics and comparative benchmarks comparing SPX against WebP and JPEG-XL across industrial datasets are available in the independent benchmark document:

👉 View Comparative Benchmarks (BENCHMARK.md)

---

8. Limitations & Roadmap

• Bit Depth: Currently limited to 8-bit per channel.
• Color Spaces: Optimized for RGB. No support for CMYK or YCbCr subsampling (G-sub transform is native to RGB).
• Alpha Channel: While RGBA is supported, the Alpha channel currently utilizes traditional Zstd compression (Level 1) rather than the high-performance rANS sharding engine used for RGB. Optimization for constant alpha (solidity detection) is a planned future improvement.
• Threading: The Python orchestration layer is single-threaded; however, the Rust-native backend utilizes internal data-parallelism via Rayon for hot-path kernels (RCT, Sharding, rANS).
• Performance Ceiling: Current throughput is achieved via branchless algorithmic design and LLVM auto-vectorization. There is no manual SIMD (AVX2/NEON) implementation. This project serves as a high-performance baseline; downstream forks seeking extreme optimizations may consider manual intrinsics or a pure-native C++/Rust port to eliminate Python orchestration overhead entirely.
• Content Bias: Benchmarks currently focus on Natural Photographic images. Validation on Synthetic Content (e.g., screenshots, UI elements, or computer graphics) is limited due to the lack of specialized testing datasets in the current pipeline. Performance on high-frequency artificial edges may vary.

9. Dataset Sources

To verify the benchmarks or test the engine with standard datasets, you can download the images from the following official sources:

• DIV2K Data Set - Train & Validation: ETH Zurich CVL

• Kodak Data Set: Kaggle - Kodak Dataset

• Clic Data Set: Kaggle - CLIC Dataset

• Tecnick Data Set: SourceForge - TestImages

• Standard Test Images (ICI): Image Compression Info

---

10. Project Background

The SPX project is a lossless image compression framework developed through a multi-phase research cycle. The project utilized the agentic AI Claude Code and Antigravity to architect technical components, including the Four-Pillar Architecture, Universal-42 Sharding, and a 4-way interleaved rANS entropy engine. In v1.0.0, the core computational kernels were migrated from Python/Numba to a Rust-native backend, achieving significantly higher throughput and reducing runtime JIT overhead.

This initiative serves as a technical proof-of-concept for AI-assisted engineering, demonstrating that autonomous agents can assist in complex algorithmic optimization and multi-language systems integration.

11. Acknowledgments

• Entropy Coding: This project utilizes the rANS algorithm developed by Dr. Jarosław (Jarek) Duda. His work on Asymmetric Numeral Systems (ANS) provided the mathematical foundation for the entropy core.
• Spatial Prediction: The spatial decorrelation engine utilizes the Median Edge Detector (MED) algorithm, originally introduced in the LOCO-I (JPEG-LS) standard.

---

Current Version: v1.0.0 | License: Apache 2.0 | MSE Target: 0.00000000