sadcompressor 0.1.3

Streamed Array Data compressor

sadcompressor

sadcompressor is a compact archival format and Python library for streamed time-series data, mainly NumPy arrays. It stores logical time keys with quantized full frames, deltas, optional prediction, dictionaries, and a tail index for fast random access.

Array compression is lossy by design. The reconstructed values are guaranteed by the configured quantization precision, not by bitwise equality with source arrays.

Quick Start

import numpy as np
import sadcompressor as sad

filename = "example.sad"

with sad.SADWriter(filename, prec_nbits=20, prec_maxexp=8) as writer:
    writer["x"] = np.array([1.0, 2.0, 3.0], dtype=np.float32)
    writer["meta"] = {"step": 0}

    writer.next_key(0.1)
    writer["x"] = np.array([1.1, 2.1, 3.1], dtype=np.float32)
    writer["meta"] = {"step": 1}

with sad.SADReader(filename) as reader:
    while not reader.next_key():
        print(f"t={reader.t:.3f}", reader["x"], reader["meta"])

Within one time key, each field can be assigned only once. Assign the same field again after writer.next_key(dt).

Random Access

SADWriter writes a tail index into the final EndFrame by default. New archives therefore open quickly through SADRandomReader without scanning all frame descriptors.

with sad.SADRandomReader(filename, decode_workers="auto") as reader:
    print(reader.nkeys)
    print(reader.timestamps)
    print(reader.list_arrays())

    reader.seek(1)
    print(reader.t, reader["x"])

For older archives without an index, SADRandomReader falls back to the old descriptor scan while skipping compressed array payloads. If the file is writable, it persists a tail index so the next random open is fast.

Use persist_index="never" to guarantee that opening for random access does not modify the file:

with sad.SADRandomReader(filename, persist_index="never") as reader:
    ...

Use index_policy="require" when tools should fail instead of scanning:

with sad.SADRandomReader(filename, index_policy="require") as reader:
    ...

SADWriter(..., write_index=False) disables writing the tail index.

Archive Inspection

open_archive() exposes the logical archive structure without reading compressed array payloads:

archive = sad.open_archive(filename, persist_index="never")

print(archive.header)
print(archive.index_status)
print(archive.structure.nkeys)
print(archive.structure.array_fields)
print(archive.structure.dict_fields)

It uses the tail index when available and falls back to descriptor scanning according to index_policy.

Compression Backends

The zlib-compatible backend is selected automatically. If isal is installed on the current platform, it is used by default; otherwise sadcompressor falls back to Python's stdlib zlib.

from sadcompressor.codec import get_zlib_backend, set_zlib_backend

set_zlib_backend("isal")      # or "stdlib", or "auto"
print(get_zlib_backend())

The same choice can be made before import with:

SADCOMPRESSOR_ZLIB_BACKEND=stdlib python script.py

Compression levels are backend-specific:

• stdlib zlib: -1..9
• isal: 0..3

with sad.SADWriter(
    filename,
    prec_nbits=20,
    prec_maxexp=8,
    compression_level=2,
    encode_workers="auto",
) as writer:
    ...

encode_workers controls parallel compression of independent array payloads. decode_workers controls parallel decoding of independent arrays in a frame. Both accept "auto" or a positive integer.

Command Line

Full user documentation: docs/cli.md.

The package installs one main entry point:

sad info example.sad
sad dump example.sad
sad copy source.sad recompressed.sad --nbits 18 --fullframe 20
sad pack -o trajectory.sad --dt 0.1 'frames/frame_*.npz'
sad extract trajectory.sad --index-range 0:100:10 -o frames

Compatibility aliases are kept:

sadinfo example.sad
sadump example.sad
sadcopy source.sad recompressed.sad

Useful copy options:

sad copy source.sad out.sad \
  --zlib-backend isal \
  --compression-level 2 \
  --encode-workers auto \
  --nbits 20 \
  --fullframe 20 \
  --prediction \
  --packbits

Benchmarks

Benchmark helpers are grouped under sad bench.

Generate a scalar 2D benchmark archive:

sad bench generate \
  --kind oscillating \
  --field u \
  --shape 512 512 \
  --frames 200 \
  --output tmp/bench_oscillating_2d.sad \
  --overwrite \
  --zlib-backend isal \
  --encode-workers auto

Generate a production-like normalized 3D vector field:

sad bench generate \
  --kind vector-normalized-3d \
  --field v \
  --shape 20 200 200 \
  --frames 200 \
  --null \
  --ui json

Read sequentially or by random access:

sad bench read-seq tmp/bench_oscillating_2d.sad \
  --field auto \
  --decode-workers auto \
  --ui json

sad bench read-random tmp/bench_oscillating_2d.sad \
  --pattern all \
  --read-fields all \
  --decode-workers auto \
  --ui json

--ui accepts auto, interactive, plain, json, or quiet.