sadcompressor 0.1.5

Streamed Array Data compressor

sadcompressor

sadcompressor stores simulation time series as compact, time-indexed archives of NumPy arrays and metadata dictionaries.

It is meant for scientific and engineering simulations where dense fields must be saved at high time resolution. In many such runs, most of the field changes slowly while small regions change quickly, for example near topological defects or sharp localized events. That locality makes the history highly compressible, but the raw .npy/.npz output can still become enormous.

SAD stores arrays on a configurable fixed-point grid. Floating-point inputs are first rounded according to the selected quantization contract; bounded contracts also clip to their declared integer range. After that, SAD compression is lossless for the quantized values. Quantized archives can therefore be copied, repacked, and recompressed without adding more error as long as the same precision is preserved. Dictionary fields are stored as UBJSON metadata.

Why Use It For Simulations

Simulation data often has three useful properties: it arrives as a stream, adjacent timesteps are related, and historical analysis can often use lower precision than the simulation used internally. SAD is built around those facts.

• Keep high time resolution. When most cells barely change between saved steps and only localized regions move quickly, SAD deltas and residual codecs can exploit that structure instead of treating every snapshot as independent.
• Write large runs as they happen. Writers emit frames incrementally, so a simulation can save a long trajectory without holding the whole history in memory.
• Read what you need. Sequential readers stream full trajectories; random readers jump to selected time keys for plotting, diagnostics, or restart-like analysis.
• Choose the analysis precision. Store the historical archive at an explicit fixed-point precision, then preserve that quantized state exactly during later repacking.
• Use Python or the shell. Write archives from NumPy code, or pack, copy, extract, compare, export, and benchmark archives with the sad CLI.
• Move data out when needed. Export to HDF5 or Zarr for external tools, then rebuild SAD archives from exported SAD layouts.
• Tune speed/size tradeoffs. Presets such as fast, balanced, and compact let you choose between quick writing and smaller archives.

Is It A Good Fit?

Use sadcompressor when:

• your data is a sequence of finite NumPy arrays over logical time;
• you can choose a fixed-point precision appropriate for later analysis;
• neighboring timesteps are correlated, especially with localized fast changes;
• you want to write and process large archives with memory bounded by the current frame or selected fields;
• you need both archive compression and later random access to selected frames;
• you want a scriptable CLI for packing snapshots, recompressing old runs, and exporting to HDF5/Zarr.

It is probably the wrong tool when:

• original floating-point values must round-trip bit-for-bit exactly;
• NaN/Inf values must be represented in array payloads;
• you need a general-purpose database or arbitrary HDF5/Zarr schema importer;
• your workload is mostly one-off, uncorrelated arrays where time-series deltas cannot help.

Quantized array writers reject NaN and Inf because the SAD payload format does not store a validity mask for non-finite values.

Performance Snapshot

Benchmarks are dataset- and machine-dependent. The real-data snapshot below uses tmp/CW_1600_consym_beta_1800.sad, a 321-key field archive with array fields ph, x, y, and z.

The archive comes from a simulation of a photoactive liquid-crystal film: x, y, and z are components of the director field, while ph is the concentration of the light isomer of a chiral dopant. This is a typical smooth-field workload: the director varies coherently in space and time, and most saved states differ by localized evolution rather than independent noise. The source SAD file is 17.59 MiB, while the decoded float32 arrays exposed by the public API are 3.13 GB and the 8-bit quantized logical arrays are 783.24 MB.

Compression comparison on that dataset:

Candidate	Time	Size	Decoded ratio	Quantized ratio
Source SAD	-	17.59 MiB	169.90x	42.48x
SAD balanced pack	9.91 s	16.60 MiB	180.00x	45.00x
SAD fast pack	9.42 s	21.56 MiB	138.61x	34.65x
HDF5 zstd15 shuffle 128MiB	39.64 s	25.31 MiB	118.07x	29.52x
Zarr zstd9 shuffle 128MiB	18.68 s	25.70 MiB	116.27x	29.07x
HDF5 zstd22 shuffle 128MiB	720.18 s	21.67 MiB	137.89x	34.47x

Read performance on the same source SAD archive:

Read mode	Payload	Time	Rate
Sequential ph	51.36 MB	1.14 s	282.7 frames/s
Sequential x	1.03 GB	1.45 s	708.4 MB/s
Random all fields, 100-frame forward window	976 MB	0.65 s	1.49 GB/s
Random all fields, 16 keyframes	156 MB	0.15 s	1.07 GB/s
Random all fields, 100 random frames	976 MB	3.80 s	257 MB/s

See benchmark results and the external archive comparison for the full tables, environment, commands, and dataset artifact notes. The real benchmark archive is intentionally kept as an external artifact rather than committed to git.

Supported Platforms

sadcompressor contains a Rust extension and currently targets Python 3.10+. Published PyPI wheels currently cover:

• Linux x86-64 glibc-based systems through manylinux2014 / manylinux_2_17;
• Windows x86-64 through win_amd64.

Other platforms are not part of the published binary wheel set yet. They can be built from a source checkout with a compatible Python, Rust, and maturin toolchain; see the Developer documentation.

Install

pip install sadcompressor

Optional features are installed with extras:

pip install 'sadcompressor[hdf5]'
pip install 'sadcompressor[zarr]'
pip install 'sadcompressor[export]'
pip install 'sadcompressor[demo]'

Python Quick Start

The example below simulates a scalar wave amplitude on a 2D periodic grid. A Gaussian pulse starts at the center, then a finite-difference wave equation spreads it across the domain. SAD stores the sequence as time keys, quantizes each grid value to 8 bits, and lets the compact preset choose frame codecs for the smooth spatial and temporal evolution.

import numpy as np
import sadcompressor as sad

# Build a small 2D wave field. This part represents your simulation state.
size = 96
frame_count = 120
axis = np.linspace(-1.0, 1.0, size, dtype=np.float32)
y, x = np.meshgrid(axis, axis, indexing="ij")
u = np.exp(-80.0 * (x**2 + y**2)).astype(np.float32)
u_prev = u.copy()

filename = "wave2d.sad"

# Open a streamed archive. Symmetric 8-bit quantization represents both
# endpoints of this [-1, 1] field.
with sad.SADWriter(
    filename,
    quantization=sad.QuantizationSpec.symmetric(nbits=8, absmax=1.0),
    compression_preset="compact",
) as writer:
    for frame in range(frame_count):
        writer["u"] = u

        # Advance the simulation after storing the current frame.
        neighbors = sum(
            np.roll(u, shift, axis)
            for shift, axis in ((1, 0), (-1, 0), (1, 1), (-1, 1))
        )
        u_prev, u = u, np.clip(
            2 * u - u_prev + 0.35**2 * (neighbors - 4 * u),
            -0.95,
            0.95,
        ).astype(np.float32)
        # Finish this archive key and advance stream time.
        writer.next_key(0.02)

# Stream the archive back. Values are dequantized to NumPy arrays on access.
with sad.SADReader(filename) as reader:
    for frame in reader:
        u = frame["u"]
        print(f"{frame.index:03d} t={frame.t:.2f} max={u.max():+.3f}")

Within one time key, each field can be assigned only once. Assign the same field again after writer.next_key(dt).

This example stores a wave amplitude field with one signed byte per grid node: QuantizationSpec.symmetric(nbits=8, absmax=1.0) maps [-1, 1] to integer codes [-127, 127], so both endpoints and zero are exactly representable. A full runnable version in demo/wave2d.py also writes the 8-bit grayscale GIF shown above. By default, generated files go under tmp/wave2d-demo/ in the repository checkout; install the demo extra for GIF output.

Running the demo with its defaults (120 frames, 96x96 grid) gives:

Representation	Size
Raw float32 arrays	4.22 MiB
8-bit quantized logical arrays	1.05 MiB
SAD compact archive	92.16 KiB
8-bit grayscale GIF	208.31 KiB

Compression Choices

Most new workflows should start with one of the stable presets:

Preset	Use it when
fast	write speed matters and you still want delta compression.
balanced	you want the default tradeoff for many simulation fields.
compact	smaller files matter more than compression time.
legacy	you need compatibility-style Q2/Q3 behavior.

Experimental presets are available under the experimental.* prefix. Users can optimize archive speed and size for their own datasets by creating custom presets; sad predictorstat --suggest-dir can generate them from measurements on real data. See predictorstat and compression presets.

Large Archives And Random Access

SAD is designed for large arrays and long time histories. SADWriter streams logical time keys as they are produced, SADReader replays the trajectory sequentially, and SADRandomReader opens selected frames for interactive tools, plotting, diagnostics, and sparse post-processing.

Random access works because built-in presets periodically write complete frames. To reconstruct a selected state, the random reader starts from the nearest complete frame and replays only the needed updates. New seekable archives also include a compact tail index, so random readers can locate indexed frames without scanning all payloads.

Sequential reading is best when an analysis consumes the whole trajectory. On the benchmark archive above, reading one x field decoded 1.03 GB in 1.45 s (708.4 MB/s). Random access is meant for different work: opening a viewer, jumping to a frame, or reading a short time window without replaying the full run. On the same archive, reading all fields for a 100-frame forward window took 0.65 s (1.49 GB/s), and reading 16 key frames took 0.15 s. Jumping to 100 unrelated frames took 3.80 s (257 MB/s) because more intermediate updates must be replayed.

with sad.SADRandomReader("wave2d.sad", decode_workers="auto") as reader:
    reader.seek(42)
    u42 = reader["u"]
    print(f"t={reader.t:.2f} max={u42.max():+.3f}")

    reader.seek(100)
    u100 = reader["u"]
    print(f"t={reader.t:.2f} max={u100.max():+.3f}")

CLI Quick Start

Pack numbered NPZ snapshots into one SAD time-series archive:

sad pack -o trajectory.sad \
  --dt 0.1 \
  --nbits 20 \
  --preset balanced \
  'frames/frame_*.npz'

Extract selected SAD frames back to NPZ:

sad extract trajectory.sad --index-range 0:100:10 -o frames

Repack a SAD archive with another compression policy:

sad pack source.sad -o recompressed.sad \
  --global-raw never \
  --nbits 18 \
  --preset compact

Export SAD for external tools:

sad export trajectory.sad trajectory.h5
sad export trajectory.sad trajectory.zarr --representation quantized --codec zstd

Pack exported HDF5/Zarr data back to SAD. This path is still experimental and is intended for SAD exports with preserved quantized metadata:

sad pack trajectory.zarr -o restored.sad

Full CLI documentation starts at docs/cli.md.

Documentation

Start with the documentation index if you are not sure which page you need.

• Concepts: archive model, precision, random access, and Python API boundaries.
• CLI: command overview with links to every command page.
• sad pack: pack snapshots and recompress SAD archives.
• sad extract: export selected frames to NPZ.
• sad export: write SAD data to HDF5 or Zarr.
• Developer documentation: project internals, builds, and releases.