s4-codec (Python bindings)

In-process CPU compression (zstd + gzip) from Python — no S4 gateway required. Wraps the same Rust s4-codec crate that powers the S4 S3-compatible storage gateway, so a Python notebook / Airflow task / Spark UDF can compress and decompress with the exact same byte format as objects sitting in an S4 bucket. (GPU codecs are intentionally NOT exposed in Python in v1.0 — they require a CUDA toolchain + GPU at runtime, which is a poor fit for pip install. Workloads that need GPU compression should route through the s4 server gateway instead. Python GPU exposure is a v1.x roadmap candidate.)

Install

pip install s4-codec        # CPU codecs only (zstd + gzip)

Example

from s4_codec import CpuZstd, CpuGzip, gpu_available

codec = CpuZstd(level=3)
data = b"hello squished s3 " * 10_000
compressed, original_size, crc = codec.compress(data)
roundtrip = codec.decompress(compressed, original_size, crc)
assert roundtrip == data

# RFC 1952 gzip output — decodable by any standard `gunzip`-aware client.
gz_compressed, *_ = CpuGzip(level=6).compress(data)
assert gz_compressed[:2] == b"\x1f\x8b"

print("GPU available:", gpu_available())

API

The (original_size, crc32c) tuple corresponds to the ChunkManifest.original_size / ChunkManifest.crc32c fields the Rust crate uses; round-trip them alongside the compressed payload (e.g. as JSON sidecar fields).

Build from source

pip install maturin
cd crates/s4-codec-py
maturin build --release
ls target/wheels/                          # *.whl is here

The --features nvcomp-gpu flag forwards to the underlying s4-codec-rs crate's GPU codecs at the Rust level, but the Python module does NOT expose Python classes for the GPU codecs in v1.0 (see the §Status note above). Building with --features nvcomp-gpu therefore only affects what gpu_available() reports, not which Python classes are importable.

maturin develop installs the wheel into the current virtualenv for iterative development.

Running tests

maturin develop
pip install -e ".[dev]"
pytest tests/

The --features nvcomp-gpu build flag forwards to the underlying s4-codec-rs GPU paths at the Rust level. In v1.0 this only affects what gpu_available() reports; the Python module does NOT add GPU codec classes when built with this feature (see the §Status note at the top of this file).

maturin develop --release --features nvcomp-gpu

The pytest suite covers CPU codec round-trips, RFC 1952 gzip compatibility, GIL-release threading, version inheritance, and the per-CodecError exception class hierarchy (v0.8.5 #85). A separate Rust-side test (tests/version_matches_workspace.rs) guards the workspace semver inherit.

Error handling

The binding raises a subclass tree per CodecError variant so callers can branch programmatically instead of string-matching:

S4Error inherits from ValueError for backward compat with code that caught the previous flat ValueError mapping. S4BackendError and S4IoError deliberately escape that hierarchy so existing retry-on-IOError middleware continues to fire on the right class.

Workspace integration

The crate ships a cdylib only and uses PyO3's extension-module feature, so cargo check -p s4-codec-py and cargo build --workspace succeed on a CI runner with no Python development headers installed — no libpython link is performed; the Python interpreter that loads the .so provides those symbols at runtime.

If you ever see a link error like undefined reference to PyExc_…, drop pyo3/extension-module from the features and you'll get the diagnostic build that does link libpython.

Threading / GIL

Both CpuZstd.compress() and CpuGzip.compress() (and their decompress() counterparts) release the Python GIL while running, so other Python threads make progress concurrently. This is safe for:

• Django / Flask workers
• ASGI / asyncio event loops (use asyncio.to_thread() to wrap the blocking call)
• multi-threaded data pipelines

Example (asyncio):

import asyncio
from s4_codec import CpuZstd

async def compress_async(data: bytes) -> bytes:
    codec = CpuZstd()
    compressed, orig_size, crc = await asyncio.to_thread(codec.compress, data)
    return compressed

Note: the methods themselves are synchronous — they don't return awaitables. The GIL release means another Python thread can run during the compress; it doesn't make the call async-aware.

Supported codecs

The GPU codecs (nvcomp-zstd, nvcomp-bitcomp, nvcomp-gdeflate) are intentionally not exposed as Python classes in v1.0 — they require a CUDA toolchain at build time and a GPU at runtime, which is a poor fit for the pip install s4-codec packaging story. The nvcomp-gpu feature on the underlying Rust crate exists for the server path; the Python module's runtime classes are the two CPU codecs above. gpu_available() -> bool is exposed for clients that want to gate their own logic on GPU presence (e.g. to decide whether to route a workload through the s4 server gateway instead of the in-process Python decoder), but it does not enable any new Python class on its own. GPU codec exposure in Python is a v1.x roadmap candidate.

Publishing status

• PyPI publish is manual (no CI automation as of v0.8.5):

  cd crates/s4-codec-py
  maturin build --release
  twine upload target/wheels/*
  

• Workspace version inheritance was fixed in v0.8.5 #82 — the published wheel version now matches the gateway version.

target/wheels/ is gitignored — never commit .whl files.

License

Apache-2.0 — same as the rest of the S4 project.