ncarnate 2.0.2

Convert HDF4/HDF-EOS2 (MODIS, AMSR-E) to CF-annotated netCDF4 and losslessly recompress netCDF/HDF5 files.

Reincarnate legacy scientific data as modern netCDF4.

ncarnate reads netCDF3, netCDF4/HDF5, and HDF4/HDF-EOS2 files and writes recompressed, CF-annotated netCDF4. It does two jobs:

• Recompress netCDF/HDF5 files — change the compression level, shuffle filter, or storage layout without changing a single stored value.

• Convert HDF4 and HDF-EOS2 granules (AMSR-E, MODIS, and kin) to netCDF4, reconstructing the CF coordinates that modern tools (xarray, QGIS, Panoply) need: grid projections become CF grid mappings with 1-D x/y and 2-D lat/lon coordinates, swath geolocation is attached as CF coordinates, and dimension-mapped (e.g. 5 km → 1 km) geolocation is interpolated through ECEF space.

Problems this solves

Reach for ncarnate if you are trying to:

• Convert HDF4 / HDF-EOS2 granules (MODIS, AMSR-E, and kin) to netCDF4 so they open cleanly in xarray, QGIS, or Panoply.

• Read an HDF-EOS2 swath or grid that has no usable lat/lon — ncarnate reconstructs CF lat/lon coordinates and grid mappings so the data is actually georeferenced, instead of an unplottable array.

• Recompress a netCDF4 / HDF5 file — change the compression level or shuffle filter without altering a single stored value.

• Shrink an archive of scientific files without risking the science: every output is verified value-for-value against its source before it replaces anything, and stored values round-trip bit-identically.

• Batch-convert a directory tree of legacy granules to modern netCDF4 in one command.

The fidelity contract

Converting or recompressing a file changes storage, never science data:

• Every variable’s stored values are preserved bit-identically — packed integers stay packed; scale_factor/add_offset/ _FillValue are carried across as declarations, never applied.

• Every dimension (including unlimited-ness), attribute (including its type), and group survives. HDF-EOS2 StructMetadata is preserved verbatim; names netCDF cannot hold are sanitized with the original recorded in a companion attribute.

• Geolocation reconstruction is strictly additive: the original information always rides along, so the conversion never becomes the only copy of the truth.

• Every output is verified against the source value-for-value before it replaces anything. A source file is never destroyed by a failed run, and HDF4 sources are never replaced at all.

• Unsupported constructs (user-defined netCDF types, unverified GCTP projections, exotic swath layouts) fail loud with a named error rather than guessing — a wrong coordinate is worse than a refused conversion. --no-geolocation converts the raw payload anyway.

The details, the guarantee boundary, and how the test suite pins each clause live in docs/fidelity-notes.md.

Installation

With conda (from conda-forge):

conda install -c conda-forge ncarnate

This works on every platform and is the recommended install on Windows — conda-forge’s pyhdf is built against a proper HDF4 library everywhere, so the full HDF4/HDF-EOS2 converter runs on Windows, macOS, and Linux alike.

With pip (from PyPI):

pip install ncarnate

On Linux (x86_64) and macOS (arm64), every dependency — including pyhdf — installs as a self-contained binary wheel with no system libraries required. On platforms without a repaired pyhdf wheel (e.g. Linux aarch64), building from sdist requires the system HDF4 library first (Debian/Ubuntu: apt install libhdf4-dev).

Windows via pip: the netCDF/HDF5 recompression path works from PyPI wheels out of the box, but the HDF4/HDF-EOS2 conversion path does not — pyhdf’s Windows wheel ships no HDF4 runtime, so import pyhdf fails with a DLL-load error. Use the conda-forge install above for HDF4 on Windows (or WSL with the pip instructions).

Command line usage

# Recompress a netCDF4 file in place (verified before replacement).
ncarnate observations.nc --complevel 9

# Keep the original; write observations_recompressed.nc beside it.
ncarnate --no-overwrite observations.nc

# Convert an HDF-EOS2 granule -> granule.nc with CF geolocation.
ncarnate AMSR_E_L3_SeaIce12km_B02_20020619.hdf

# Convert the raw SDS payload only (unsupported-projection escape hatch).
ncarnate --no-geolocation granule.hdf

# Recurse over a directory tree.
ncarnate -r /data/archive

Exit codes: 0 success, 1 one or more files failed, 2 bad input paths or arguments.

Library usage

from ncarnate import recompress

# Lossless recompression; returns the output path.
recompress("observations.nc", complevel=9)

# HDF-EOS2 conversion; the .hdf source is never replaced.
recompress("granule.hdf", dst="granule.nc")

Example

The AMSR-E daily 12.5 km sea-ice granule this project grew up around:

File	Input	Output
netCDF4 recompression (--complevel 9)	42.6 MB	19.9 MB
HDF-EOS2 → netCDF4 (+ reconstructed lat/lon)	60.2 MB	35.5 MB

Both outputs re-read bit-identically to their sources; the conversion additionally carries CF polar_stereographic grid mappings and coordinates for both hemispheric grids. The northern grid’s reconstructed latitudes/longitudes agree with The HDF Group’s independent conversion of the same granule to within 10^-5 degrees (about a metre), the tolerance the test suite enforces.

Supported inputs

• netCDF4 / HDF5 and netCDF3 — recompressed via the netCDF4 library.

• HDF4 / HDF-EOS2 — read via the pyhdf SD API. GRID structures with GCTP polar-stereographic, geographic, and Lambert-azimuthal (EASE-Grid) projections; SWATH structures with direct or dimension-mapped geolocation. Output is always netCDF4 — HDF4 is never written.

Development

pip install -e ".[test]"
ruff check .
pytest

The test suite runs entirely offline against small committed fixtures trimmed from real granules (provenance sidecars included); cross-checks against the raw multi-MB granules self-skip where the local granule store is absent.

License

MIT — see LICENSE.