earthframe-codec

earthframe-codec is a zero-runtime-dependency Python library for bounded, incremental inspection of environmental binary frames. It is designed for telemetry gateways and archive readers that cannot load multi-gigabyte products into memory just to validate a header.

The package exposes one status contract for every input buffer:

• SCIENCE_VALID_FRAME: structural checks passed.
• SCIENCE_INCOMPLETE_STREAM: more bytes are required.
• SCIENCE_CORRUPTED_FRAME: the bytes cannot satisfy the selected format rules.

Scope

The current release provides production-safe structural decoders for GRIB2, BUFR, HDF5 signatures/superblocks, and NEXRAD archive messages. GRIB2 simple packing, BUFR descriptor words, and legacy one-byte radar bins have reusable bitwise primitives. HDF5 object trees and vendor-specific radar message payloads intentionally remain exposed as bounded raw offsets because their layouts depend on version, templates, filters, and local tables.

This is not a claim that one decoder can infer every vendor template or every NetCDF convention. NetCDF4 data is HDF5 storage with NetCDF metadata conventions; callers should use the returned HDF5 superblock information to schedule dataset-specific block reads.

Installation

python -m pip install earthframe-codec

No NumPy, HDF5 C library, or network service is required at runtime. Python 3.10+ is supported.

Streaming integration

from earthframe_codec import FrameDecoder, FrameStatus

reader = FrameDecoder(max_frame_size=256 * 1024 * 1024)
with open("telemetry.bin", "rb", buffering=0) as source:
    while block := source.read(1024 * 1024):
        for result in reader.feed(block):
            if result.status is FrameStatus.SCIENCE_VALID_FRAME:
                print(result.protocol, result.consumed, result.data.keys())
            elif result.status is FrameStatus.SCIENCE_CORRUPTED_FRAME:
                raise ValueError(result.errors)
    for result in reader.feed(b"", final=True):
        print(result.status, result.errors)

FrameDecoder retains a bytearray only for the current incomplete record. decode_frame(memoryview(...)) accepts a caller-owned buffer and does not copy section payloads while validating them. Results copy only small metadata fields; raw payloads are included as bytes where a protocol has no safe universal template.

Protocol fields

GRIB2

The decoder verifies GRIB, edition 2, the 8-byte total length, section lengths, section ordering boundaries, and the 7777 end marker. Section summaries expose raw section bytes plus common fields:

from earthframe_codec.grib2 import unpack_simple_packing
values = unpack_simple_packing(payload, count=nx * ny, bits_per_value=12,
                               reference=reference_value,
                               binary_scale=binary_scale,
                               decimal_scale=decimal_scale)

Grid schemas are returned as metadata rather than guessed coordinates. Common section 3 fields include grid_template, ni, and nj; use the template-specific Earth model and scan-mode flags to reconstruct longitude/latitude arrays.

BUFR

BUFR section boundaries and edition are checked. Section 3 descriptor words are decoded into (F, X, Y) using F=(word >> 14) & 3, X=(word >> 8) & 63, and Y=word & 255. Table B/C/D interpretation is intentionally supplied by the application or station profile because descriptors and local tables evolve independently of the wire framing.

HDF5 / NetCDF4

The HDF5 signature and superblock version/address sizes are checked without reading the full file. root_object_offset, when present in superblock v0, is an absolute file offset suitable for a block scheduler. Dataset filters, fractal heaps, B-trees, and NetCDF attributes must be interpreted against the file's exact HDF5 version and are not safely interchangeable.

NEXRAD

The archive signature, message length, type, and channel are checked. decode_legacy_bins turns byte-valued REF/VEL/SW bins into scaled values while preserving the configured missing sentinel. Vendor message 1/31/packet-structure templates should be decoded by a product-specific adapter using the returned payload.

Development

python -m pip install -e ".[test]"
python -m pytest

The GitHub Actions workflow builds and publishes distributions for version tags matching v*.*.*. Configure PyPI trusted publishing for the repository before pushing a tag.