zxtd

High-performance Zstandard compression/decompression library with dictionary support, multi-threading, and Python bindings.

Author: Quadrium

Features

• Fast compression/decompression using Facebook's Zstandard (zstd) library
• Dictionary support for better compression ratios on small data
• Multi-threaded compression for improved performance on large data
• Streaming API for memory-efficient processing of large files
• Frame inspection to detect dictionary IDs and content sizes
• Checksum verification for data integrity
• Python bindings via PyO3/maturin
• Async support (optional feature)

Installation

pip install zxtd

Python Usage

import zxtd

# Basic compression/decompression
data = b"Hello, Zstandard! " * 100
compressed = zxtd.compress(data, 3)  # level 1-22
decompressed = zxtd.decompress(compressed)
assert decompressed == data

# Dictionary compression (better for small, similar data)
dict_data = b"common prefix that repeats often "
dictionary = zxtd.Dictionary.from_bytes(dict_data)

original = b"common prefix that repeats often - unique part"
compressed = zxtd.compress_with_dict(original, 3, dictionary)
decompressed = zxtd.decompress_with_dict(compressed, dictionary)

# Custom compression parameters
params = zxtd.CompressionParams(6)
params.with_threads(4)           # Use 4 threads
params.with_checksum(True)       # Enable checksum
compressed = zxtd.compress_with_params(data, params)

# Frame inspection
info = zxtd.frame_info(compressed)
print(f"Magic: 0x{info.magic:x}, Dict ID: {info.dict_id}")

# Detect dictionary ID from frame
dict_id = zxtd.Dictionary.detect_dict_id(compressed)

# Compression levels
levels = zxtd.compression_levels()
# {'FASTEST': 1, 'FAST': 3, 'DEFAULT': 3, 'BETTER': 6, 'BEST': 9, 'ULTRA': 22}

# Verify checksum
decompressed = zxtd.decompress_verify(compressed)

Rust Usage

Add to Cargo.toml:

zxtd = { path = "../zstd-decompress", features = ["async"] }

use zxtd::{compress, decompress, CompressionParams, Dictionary};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let data = b"Hello, Zstandard!";
    
    // Basic usage
    let compressed = compress(data, 3)?;
    let decompressed = decompress(&compressed)?;
    assert_eq!(data, &decompressed[..]);
    
    // With dictionary
    let dict = Dictionary::new(b"common prefix ".to_vec())?;
    let compressed = compress_with_dict(data, 3, &dict)?;
    let decompressed = decompress_with_dict(&compressed, &dict)?;
    
    // Custom params with multi-threading
    let params = CompressionParams::new(6).with_threads(4);
    let compressed = compress_with_params(data, &params, None)?;
    
    // Streaming
    let mut decompressor = StreamingDecompressor::new(compressed)?;
    let mut buf = [0u8; 1024];
    while let Ok(n) = decompressor.read(&mut buf) {
        if n == 0 { break; }
        // process buf[..n]
    }
    
    Ok(())
}

Performance

• Multi-threaded compression scales linearly with cores
• Dictionary compression can achieve 2-5x better ratios on small similar data
• Streaming API uses constant memory regardless of input size
• Zero-copy decompression where possible

License

MIT