drb-chunk-netcdf 0.1.0

Physical chunk model for the drb:netcdf / drb:hdf classes

drb-chunk-netcdf

netCDF/HDF5-backed physical chunk model for the drb:netcdf / drb:hdf classes, built on top of the drb-chunk add-on.

What it is

drb-chunk-netcdf teaches drb-chunk how to read a netCDF-4/HDF5 variable on its native chunk boundaries instead of on whatever grid a product descriptor happens to declare. It targets the generic drb:netcdf/drb:hdf topic classes, not a single product family — a format chunk add-on, sibling to drb-chunk-zarr and drb-chunk-image (GDAL rasters).

Concretely, it registers a PhysicalTilingProvider (netcdf-chunks) that probes a variable's native chunking via netCDF4, without loading the full variable. Classic netCDF-3 variables (and any HDF5 variable stored contiguously) have no native chunk grid; the provider falls back to the whole-variable shape as a single physical chunk.

How it self-activates

Installing the package is enough; there is no product-side wiring:

• It registers NetcdfChunksProvider under the drb.chunk.physical entry-point group (drb.addons.chunk.netcdf.provider:NetcdfChunksProvider), the seam the drb-chunk core uses to discover physical-tiling providers lazily, and a matching netcdf-var physical reader via register_physical_reader.
• It ships a cortex.ttl carrying two triples — <http://www.gael.fr/drb#netcdf> drb:physicalTiling "netcdf-chunks" and <http://www.gael.fr/drb#hdf> drb:physicalTiling "netcdf-chunks" — declared under the drb.topic entry-point group. The core's ManagerDao merges every installed drb.topic descriptor into one RDF graph, so both triples land on the existing drb:netcdf and drb:hdf topics without touching the driver or any other package. Both topics share the same drb-driver-netcdf implementation, so one provider serves both.

Once both are installed, any chunk descriptor whose drb:source resolves to a netCDF/HDF5 variable node automatically gets a physical chunk model: the core resolves the source's format topic, reads drb:physicalTiling, looks up netcdf-chunks, and calls probe().

A netCDF/HDF5 variable node (DrbNetcdfVariableNode) does not itself classify as drb:netcdf/drb:hdf — the format topic matches the top-level file node, an ancestor. The core's _physical_model() walks source_node.parent up from the variable until an ancestor resolves to a format topic declaring drb:physicalTiling, so this works with the variable node obtained through the driver's normal navigation — no special-casing needed in this add-on.

The two-layer model

drb-chunk separates the physical chunk model — the source's native chunking, discovered (never declared), the unit of I/O this add-on provides — from the logical chunk model — the output representation a product chunk declares (a regular N-D grid today; burst, and later healpix, for other formats). A chunk descriptor with no declared logical geometry (drb:chunkShape/drb:transformer) simply adopts the physical grid as its logical grid: chunk.array.transformer becomes a RegularGrid whose chunk_shape is the variable's native chunking. A descriptor that declares its own grid (e.g. a coarser or finer regular tiling than the native chunks) keeps that logical grid, but reads are still issued on the physical, native-chunk-aligned boundaries and remapped into the requested logical window — so adjacent logical tiles that share a native chunk only pay for that chunk once (coalesced, cached per source node). probe() also accepts a multiplier (default 1), read from a chunk's drb:nativeMultiplier, to request an exact multiple of the native chunk as the physical unit.

netCDF-3 / contiguous fallback

Classic netCDF-3 files (and HDF5 variables explicitly stored contiguously) have no native chunk grid at all: netCDF4.Variable.chunking() returns None (or the string "contiguous") rather than a shape tuple. In that case probe() falls back to the whole variable as a single physical chunk (chunk_shape == array_shape) — correct, if coarse: any read touching the variable pulls the whole thing in once, cached for subsequent windows, rather than failing or guessing a tiling that isn't actually there.

Installation

pip install drb-chunk-netcdf

Requires drb-chunk>=0.6.1,<1, netCDF4 and numpy (installed as dependencies), and drb-driver-netcdf to resolve drb:netcdf/drb:hdf sources.

Usage

No API of its own to call — once installed alongside drb-chunk, any product add-on's Chunk.select(...) over a netCDF/HDF5-backed source picks up the physical model transparently:

from drb.topics import resolver
from drb.addons.addon import AddonManager
from drb.chunk.selection import WindowSelection

topic, node = resolver.resolve("/path/to/product.nc")
addon = AddonManager().get_addon("chunk")

chunk = addon.apply(node, chunk_name="band_04")   # e.g. a product add-on
window = WindowSelection(x=0, y=0, w=128, h=128)
array = chunk.select(window).get_impl(numpy.ndarray)

Reads above are issued on the variable's native chunk boundaries (e.g. 1x64x64), cached per source node, and remapped into the requested window — not read once per requested tile.

N-D note

netCDF/HDF5 chunking is natively N-D (Variable.chunking()/.shape cover every dimension, not just two spatial axes); the core's RemapEngine and aligned_window/coalescing logic are already N-D-general, so this provider handles variables of any rank as-is — a leading time/band axis is just another dimension in the chunk grid, not a special case.

Limitations

• Read-only probing: this add-on discovers the native chunking, it does not write or re-chunk files.
• drb:source must resolve/navigate to the variable node itself (the node whose get_impl(netCDF4.Variable) yields the variable) — the same convention drb-chunk-zarr/drb-chunk-image use for the array/raster.
• The netCDF-3/contiguous fallback treats the whole variable as one physical chunk; there is no partial native tiling to discover for those formats.

Reference

See docs/dev/writing-a-physical-tiling-provider.md in drb-chunk for the PhysicalTilingProvider contract; drb-chunk-image is the reference implementation, drb-chunk-zarr the second, this add-on the third.