drb-chunk

DRB N-D lazy chunk access add-on. It exposes N-dimensional, lazily-read data chunks (raster tiles, windows, byte ranges, time-series blocks) as first-class DRB objects, driven declaratively from a topic's RDF/Turtle descriptor and materialised on demand through numpy or a dask-backed xarray.

The add-on reads nothing until you ask for bytes. Selection (select, __getitem__) only narrows a manifest; the actual I/O happens when you call get_impl(...) / to_xarray(), always through a coalesced, physically-aligned window read (RemapEngine) — the add-on refuses to silently fall back to a full read.

pip install drb-chunk

Requires Python 3.11–3.13. The core itself has no format dependency (no rasterio, no GDAL): numpy, dask and xarray back lazy assembly; the actual content read comes from a format add-on's leaf reader (e.g. drb-chunk-image pulls in rasterio), needed only at materialisation time.

---

Why it exists

DRB drivers already turn a file or URL into a tree of DrbNodes. But a single raster band or a large array is not naturally a "node" — it is an N-D grid you want to slice cheaply, possibly remotely, without loading the whole thing.

drb-chunk adds that missing layer:

• Declarative — a topic says, in its cortex.ttl, what its chunks are (dims, dtype, transformer, where the bytes live). No code per product.
• Lazy — narrowing a chunk is pure metadata arithmetic over the transformer. I/O is deferred to materialisation.
• Driver-reusing — a format's leaf reader reads through the source node's existing get_impl (e.g. rasterio.io.DatasetReader). No driver is modified to support chunking.

---

Core concepts

Object	Role
ChunkAddon	Entry point (drb.addon = chunk). Reads a topic's descriptors and builds Chunks from a node.
ChunkDescriptor	The parsed drb:chunk declaration: name, source, dims, dtype, transformer, optional selection/collection.
Chunk	A handle on one declared chunk of a node. Slice it, then materialise it.
ChunkArray	Pure metadata of the array: dims, shape, dtype, transformer.
Transformer / WindowingTransformer / RegularGrid	The user-facing representation layer: geometry + how a Selection resolves into tile keys.
ChunkManifest	Lazy Mapping[key -> ChunkRef] over the transformer's keys.
ChunkRef	One tile's locator: key, source node, and a window or byte_range.
Selection	Serialisable, declarative constraint (isel, sel, window, band, range).
RemapEngine	Internal mechanics: coalesces a logical window onto physical-chunk boundaries, reads once, caches, crops.
PhysicalChunkModel / PhysicalTilingProvider	The source's native chunking (unit of I/O), discovered from a format add-on, never declared in a product descriptor.

Data flow:

topic (cortex.ttl)                       node (DrbNode)
      │  drb:chunk …                            │
      ▼                                          │
ChunkDescriptor ──ChunkAddon.apply(node)────────►  Chunk (Transformer + RemapEngine)
                                                   │  .select(Selection)   (lazy: subsets the manifest)
                                                   ▼
                                                 Chunk (narrowed)
                                                   │  .get_impl(np.ndarray) / .to_xarray()
                                                   ▼
                                           RemapEngine.materialize(window)  ──►  numpy / xarray
                                           (coalesced read via the format's leaf reader)

---

Architecture

The add-on is organised in three layers, each ignoring the details of the one below it and each with its own extension seam. See docs/dev/architecture-layers.md for the full write-up; the summary:

┌─────────────────────────────────────────────────────────────────────┐
│  TRANSFORMER  (user-facing, drb/chunk/transformer.py)                 │
│  Transformer / WindowingTransformer / RegularGrid                    │
│  TransformerProvider  ← entry point drb.chunk.transformer              │
│  descriptor.py  ChunkDescriptor + retrieve_chunks()  ←── cortex.ttl   │
│  selection.py   Selection (isel/window/band/range/sel) + aggregator   │
│        "what geometry a chunk presents; how a Selection becomes keys"  │
├─────────────────────────────────────────────────────────────────────┤
│  LAZINESS (metadata only, zero bytes read)                            │
│  model.py       ChunkArray, ChunkManifest, ChunkRef                   │
│  chunk.py       Chunk  (select / __getitem__ / tiles / locator)       │
│        "a handle you narrow; a lazy key→ref manifest"                  │
├─────────────────────────────────────────────────────────────────────┤
│  MECHANICS  (internal, not user-facing, drb/chunk/engine.py)          │
│  RemapEngine   ← built via BuildContext.build_engine                  │
│  remap.py       aligned_window / plan_remap / materialize              │
│  cache.py       BytesLRU (per-source region cache) + plan_cache        │
│        "coalesce a logical window onto physical-chunk boundaries,      │
│         read once, cache, crop"                                        │
├─────────────────────────────────────────────────────────────────────┤
│  PHYSICAL  (format add-on, drb/chunk/physical.py)                     │
│  PhysicalChunkModel(chunk_shape, reader, array_shape)                 │
│  PhysicalTilingProvider  ← entry point drb.chunk.physical               │
│  leaf reader  ← register_physical_reader(token, fn)                   │
│  chunk.py       get_impl(numpy) / to_xarray()  + interop.to_kerchunk  │
│        "the source's native chunking; how to read one native block"    │
└─────────────────────────────────────────────────────────────────────┘

The guiding rule, enforced in the code: nothing is read before materialisation — select() is pure key arithmetic; RemapEngine.materialize() (via the format's leaf reader) is the first place that touches the source.

Declaration & discovery

ChunkAddon (core.py) is a singleton registered under the drb.addon entry point; DRB loads it into AddonManager. It implements the Addon contract (identifier, return_type, can_apply, apply). Its core job is the build pipeline behind apply():

topic ──retrieve_chunks()──► {name: ChunkDescriptor}
                                   │  for the requested chunk:
cd.source.extract(node) ───────────┤  where are the bytes?
_resolve_source() ─────────────────┤  "." → the node itself ; else resolver.create(url)
_build_engine(source_node) ────────┤  physical model + RemapEngine (or None)
                                   ▼
ChunkArray(dims, shape, dtype, transformer)
RegularGridManifest(array, source_node)
                                   ▼
       Chunk(name, array, node, manifest, engine, topic_uri)

retrieve_chunks() (descriptor.py) inherits descriptors through rdfs:subClassOf (recursing into parents) then overrides them with the topic's own chunks, keyed by drb:chunkName — the same pattern as MetadataAddon. It reads the public RDF graph exposed by the topic's ManagerDao.

A descriptor-level drb:selection is rejected at build time; apply selections explicitly via Chunk.select().

Transformer — geometry and selection

This is "the only place grid geometry lives" for a windowing transformer (WindowingTransformer docstring). Two responsibilities: enumerate tile keys (keys(array)) and resolve a selection into keys + residual (resolve(selection, array) → ResolvedSelection). RegularGrid is the one built-in implementation:

grid_shape = ceil(shape / chunk_shape)   per dim
keys       = itertools.product(range(n) for n in grid_shape)

resolve(WindowSelection(x,y,w,h)):
    x → [x, x+w)   y → [y, y+h)
    per dim:  first = start // chunk ;  last = (stop-1) // chunk
    keys = product(range(first, last+1) ...)

Selection (selection.py) is deliberately pure, serialisable data (to_dict() / parse_selection()) — it does not know how to turn itself into keys; that is the transformer's job. This separation is what makes a Chunk serialisable (locator) and a selection replayable. SelectionAggregator composes several per-dimension constraints. A content-driven transformer (geometry only known from the product, e.g. Sentinel-1 burst) is supplied by an add-on's TransformerProvider — see docs/dev/writing-a-transformer.md.

Laziness

• ChunkArray — pure metadata (dims, shape, dtype, transformer). No bytes.
• ChunkManifest — a lazy Mapping[key → ChunkRef]. RegularGridManifest.ref(key) computes the window on the fly (key * chunk_shape); it stores nothing.
• ChunkRef — one tile's locator: key, source node, and either a window (the mainstream, format-native path) or a byte_range (used by to_kerchunk export, see Locators and interop below).
• Chunk.select() — laziness in action: resolve the selection into keys, then manifest.subset(resolved) returns a _SubsetManifest keeping only those keys, wrapped in a new immutable Chunk. No read happens.

Mechanics — RemapEngine

Materialisation is the only moment bytes are read, and it always goes through a RemapEngine (drb/chunk/engine.py) bound to the chunk's source node — internal plumbing, not a transformer, not user-facing, never constructed by hand. For a requested logical window, RemapEngine.materialize:

aligned_window(window, phys_chunk_shape, array_shape)  →  physically-aligned region
BytesLRU.get_or_read((source_id, region), lambda: leaf_reader(node, region))
crop the coalesced region down to the requested window

so adjacent logical tiles sharing one native (physical) block only pay for that block once. A RemapEngine is built once per source via BuildContext.build_engine, which resolves the source's PhysicalChunkModel (next layer) and returns None when the format declares none — in which case the chunk has no engine and cannot be materialised (enumeration, locator() and to_kerchunk still work without one).

Physical — native chunking and the leaf reader

PhysicalTilingProvider.probe(node, multiplier=1) (a format add-on) opens the source through its existing get_impl and returns a PhysicalChunkModel (chunk_shape, reader token, array_shape) describing its native chunking — discovered from the implementation, never declared in a product descriptor. The paired leaf reader, fn(node, phys_window) -> ndarray registered via register_physical_reader(token, fn), is the only content-reading path for that format: the core ships none itself and never imports a format library. See docs/dev/writing-a-physical-tiling-provider.md.

From Chunk there are two exits: get_impl(np.ndarray) (single-tile direct read through the engine) and to_xarray() (dask-backed da.from_delayed(dask.delayed(engine.materialize)), dims ("band",) + array.dims). No driver is modified to support chunking.

Design decisions to remember

Decision	Why
Selection = pure data, resolution = transformer	makes Chunk serialisable and selections replayable; one home for geometry
Lazy manifest (ref() computes the window)	no materialised tile list for very large grids
Physical model discovered, not declared; leaf reader is the only content-reading path	zero driver changes; chunking is an additive layer, format libraries stay out of the core
Reads always coalesced onto physical-chunk boundaries via RemapEngine	adjacent logical tiles sharing a native block pay for it once
No fallback to a full read	guarantees a chunk stays a chunk (no accidental full read)
Explicit v1 deferrals with clear errors	bounded scope: regular transformer + provider-supplied windowing transformers only, single-tile materialisation; drb:selection in descriptor rejected

---

Declaring chunks in a topic (cortex.ttl)

Chunks are declared on a DrbTopic with the drb:chunk predicate. Each drb:chunk blank node describes one chunk. Descriptors are inherited through rdfs:subClassOf and a child topic may override a parent's chunk by reusing its drb:chunkName.

@prefix owl:  <http://www.w3.org/2002/07/owl#> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .
@prefix drb:  <http://www.gael.fr/drb#> .

drb:raster-base a owl:Class ;
    rdfs:label "raster-base" ;
    drb:chunk [
        drb:chunkName    "data" ;
        drb:source       "." ;          # bare literal: the node itself
        drb:dims         ( "y" "x" ) ;
        drb:dtype        "uint16" ;
        drb:transformer  "regular" ;    # optional: "regular" is the default
        drb:tileHeight   512 ;
        drb:tileWidth    512
    ] .

drb:my-image a owl:Class ;
    rdfs:label "my-image" ;
    rdfs:subClassOf drb:raster-base ;   # inherits "data", adds "b04"
    drb:chunk [
        drb:chunkName    "b04" ;
        drb:source       [ drb:xquery
            "GRANULE/*/IMG_DATA/R10m/*[fn:matches(fn:name(),'.*_B04_10m\\.jp2$')]"
        ] ;                             # typed blank node: XQuery navigates the product
        drb:dims         ( "y" "x" ) ;
        drb:dtype        "uint16" ;
        drb:chunkShape   ( 256 256 )    # equivalent to tileHeight/tileWidth
    ] .

drb:chunk vocabulary

Predicate	Meaning	Required
drb:chunkName	Unique chunk identifier within the topic.	yes
drb:source	Where the bytes are. Bare literal: "." / "" = the node itself; a path/URL resolved through DRB (ConstantExtractor). Typed blank node: [ drb:xquery "…" ] — an XQuery evaluated against the product node, returning the band DrbNode (XQueryExtractor). Also accepts drb:python, drb:script, drb:constant via parse_extractor. Note: drb's XQuery engine uses full-match semantics, so patterns must carry a leading .* (e.g. fn:matches(fn:name(),'.*_B04_10m\\.jp2$')).	yes
drb:dims	RDF list of dimension names, e.g. ( "y" "x" ).	yes
drb:dtype	NumPy dtype string, e.g. "uint16".	yes
drb:transformer	Transformer name (drb:transformer token). Defaults to "regular" when omitted and the chunk declares a drb:chunkShape/drb:tileHeight+drb:tileWidth; omit entirely (no shape either) to delegate the grid to the source's physical chunk model. Non-"regular" values dispatch to a TransformerProvider (see docs/dev/writing-a-transformer.md). Can be overridden per-build via apply(transformer=…).	no
drb:chunkShape	RDF list giving the tile shape per dim.	one of chunkShape / (tileHeight + tileWidth) / delegated to the physical model
drb:tileHeight, drb:tileWidth	Convenience for 2-D (y, x) grids; equivalent to drb:chunkShape ( h w ).
drb:nativeMultiplier	Integer scaling the source's native (physical) chunk unit up before it drives reads (e.g. read 2×2 native tiles as one physical chunk). Defaults to 1.	no
drb:collection	Logical group name. Chunks sharing the same drb:collection value can be built together with apply(collection=…).	no
drb:selection	v1 deferral — a descriptor-level default selection is rejected at build time; apply selections explicitly via Chunk.select().	no

The legacy drb:tilingScheme / drb:reader predicates are no longer supported: a descriptor still carrying either raises DrbChunkError at load time ("upgrade the descriptor to drb:transformer"). Reading is now always mediated by the format's registered physical leaf reader through RemapEngine, not a per-chunk reader hint.

Collections

A collection groups related chunks under a shared name so callers can build all of them in one call. Declare it with drb:collection on each chunk that belongs to the group:

drb:sentinel2-l2a-r10m a owl:Class ;
    rdfs:label "sentinel2-l2a-r10m" ;
    drb:chunk [
        drb:chunkName  "B02" ;
        drb:source     [ drb:xquery
            "GRANULE/*/IMG_DATA/R10m/*[fn:matches(fn:name(),'.*_B02_10m\\.jp2$')]"
        ] ;
        drb:dims       ( "y" "x" ) ;
        drb:dtype      "uint16" ;
        drb:tileHeight 512 ; drb:tileWidth 512 ;
        drb:collection "R10m"
    ] ;
    drb:chunk [
        drb:chunkName  "B04" ;
        drb:source     [ drb:xquery
            "GRANULE/*/IMG_DATA/R10m/*[fn:matches(fn:name(),'.*_B04_10m\\.jp2$')]"
        ] ;
        drb:dims       ( "y" "x" ) ;
        drb:dtype      "uint16" ;
        drb:tileHeight 512 ; drb:tileWidth 512 ;
        drb:collection "R10m"
    ] .

ChunkAddon exposes two collection-aware API calls:

# Map collection name -> list of chunk names (None key = ungrouped chunks).
chunk_addon.available_collections(topic)
# {'R10m': ['B02', 'B04'], None: ['QI']}

# Build every chunk in a collection.
chunks = chunk_addon.apply(node, collection="R10m", topic=topic)
# -> [Chunk("B02"), Chunk("B04")]

apply(collection=…) raises DrbChunkError listing available collections if the requested name is unknown. chunk_name and collection are mutually exclusive — passing both raises DrbChunkError.

---

Quick start (through the DRB resolver)

from drb.topics import resolver
from drb.addons.addon import AddonManager

# Resolve any source DRB knows how to type.
topic, node = resolver.resolve("/data/S2/IMG_DATA/T31TCJ_B04.jp2")

chunk_addon = AddonManager().get_addon("chunk")   # the registered singleton

if chunk_addon.can_apply(topic):
    # What chunks does this topic declare?
    for name, transformer in chunk_addon.available_chunks(topic):
        print(name, transformer)       # ('data', {'regular': {'chunk_shape': [512, 512]}})

    # What collections are declared? (None key = ungrouped chunks)
    print(chunk_addon.available_collections(topic))
    # {'R10m': ['B02', 'B04'], None: ['QI']}

    # Build one named chunk (or omit chunk_name to get a list of all chunks).
    chunk = chunk_addon.apply(node, chunk_name="data")

    # Build every chunk in a collection.
    chunks = chunk_addon.apply(node, collection="R10m")

apply() returns a single Chunk when chunk_name is given, a list[Chunk] when collection is given, or a list[Chunk] of every declared chunk when neither is given. It raises DrbChunkError if the topic declares no chunk, if chunk_name is unknown, if collection is unknown, or if both are given.

---

Working with a Chunk

Inspect the grid

chunk.name              # "data"
chunk.array.dims        # ("y", "x")
chunk.array.shape       # (10980, 10980)   — inferred from the source raster
chunk.array.dtype       # "uint16"
chunk.grid_shape        # (22, 22)         — ceil(shape / chunk_shape), RegularGrid only

list(chunk.tiles())     # [(0, 0), (0, 1), …]   — tile keys
chunk.tile((0, 0))      # ChunkRef(key=(0,0), source=…, window=((0,512),(0,512)))

Select (lazy — reads nothing)

select resolves the selection into tile keys via the transformer and returns a new Chunk over the narrowed manifest. chunk[sel] is sugar for chunk.select(sel).

from drb.chunk.selection import WindowSelection, IselSelection, BandSelection

# A pixel window (x, y, w, h); maps to the x/y dims of a RegularGrid.
roi = chunk.select(WindowSelection(x=512, y=0, w=512, h=512))

# Integer-position selection per dim: int -> single index, [start, stop] -> range.
sub = chunk.select(IselSelection({"y": [0, 1024], "x": [0, 512]}))

Materialise (this is where I/O happens)

import numpy as np

# Single tile -> numpy. Multi-tile numpy materialisation is rejected on purpose.
arr = roi.get_impl(np.ndarray)          # windowed read via RemapEngine, shape (bands, h, w)

# Lazy, dask-backed xarray.DataArray (dims = ("band",) + array.dims).
xda = roi.to_xarray()
xda = roi.get_impl(__import__("xarray").DataArray)   # equivalent

get_impl(np.ndarray) requires the selection to resolve to a single tile — otherwise it raises and points you to to_xarray(). (In v1, to_xarray() itself also only assembles a single tile; see Limitations.)

---

More usage examples

Manual construction with a RemapEngine

A Chunk materialises through a RemapEngine bound to its source node — a Chunk built without one (engine=None) still supports enumeration, select() and locator(), but raises DrbChunkError on get_impl/to_xarray. Building both by hand (no topic/descriptor involved) mirrors what ChunkAddon._build does internally (see docs/dev/architecture-layers.md); the windowed read returns exactly the same data as a full slice, without loading the whole source, and adjacent tiles sharing a native (physical) block only pay for it once (proven in tests/test_chunk.py and tests/test_engine.py):

import numpy as np

from drb.chunk.cache import BytesLRU
from drb.chunk.chunk import Chunk
from drb.chunk.engine import RemapEngine
from drb.chunk.model import ChunkArray, RegularGridManifest
from drb.chunk.physical import PhysicalChunkModel, register_physical_reader
from drb.chunk.selection import WindowSelection
from drb.chunk.transformer import RegularGrid

# The leaf reader is the format add-on's only content-reading path; here a
# stand-in for e.g. drb-chunk-image's rasterio-backed "raster" reader.
def read_block(node, phys_window):
    (y0, y1), (x0, x1) = phys_window
    return node.get_impl(np.ndarray)[:, y0:y1, x0:x1]

register_physical_reader("my-reader", read_block)

model = PhysicalChunkModel(chunk_shape=(512, 512), reader="my-reader",
                           array_shape=(1024, 1024))
engine = RemapEngine(node, model, (1024, 1024), cache=BytesLRU(64 * 2**20))

array = ChunkArray(dims=("y", "x"), shape=(1024, 1024), dtype="uint16",
                   transformer=RegularGrid(chunk_shape=(512, 512)))
chunk = Chunk(name="data", array=array, node=node,
              manifest=RegularGridManifest(array, node), engine=engine)

out = chunk.select(WindowSelection(x=512, y=0, w=512, h=512)).get_impl(np.ndarray)
# out == full_data[:, 0:512, 512:1024]

In practice you rarely construct a RemapEngine yourself: ChunkAddon.apply() builds it from the topic's drb:chunk declaration and the source's drb:physicalTiling, and a TransformerProvider.build() gets one through ctx.build_engine(...).

Lazy xarray / dask compute

xda = roi.to_xarray()              # dask-backed DataArray, dims ("band", "y", "x")
result = xda.mean().compute()      # the deferred read fires only on .compute()

---

Selection types

All selections are pure, serialisable data (to_dict() / parse_selection()); turning them into tile keys is the transformer's job (WindowingTransformer.resolve).

Type	to_dict() shape	Resolvable by RegularGrid in v1
IselSelection(per_dim)	{"isel": {"y": [0,1024], "x": 5}}	✅
WindowSelection(x,y,w,h)	{"window": {"x":…, "y":…, "w":…, "h":…}}	✅ (maps to x/y dims)
SelSelection(per_dim)	{"sel": {…}} (label/coord based)	❌ (needs coords)
BandSelection(bands)	{"band": [0, 3]}	❌
RangeSelection(offset,length)	{"range": {"offset":…, "length":…}}	❌
SelectionAggregator(parts)	merged keys of its parts	per-part

Round-tripping:

from drb.chunk.selection import parse_selection

sel = parse_selection({"window": {"x": 0, "y": 0, "w": 256, "h": 256}})
sel.to_dict()           # {'window': {'x': 0, 'y': 0, 'w': 256, 'h': 256}}

# Several keys -> a SelectionAggregator composing leaf constraints.
parse_selection({"isel": {"y": [0, 512]}, "band": [0]})

Out-of-bounds or unknown-dimension selections raise DrbSelectionError.

---

Materialisation: RemapEngine and the physical leaf reader

A Chunk materialises through the RemapEngine bound to its source node (Chunk(engine=...)), not through a per-chunk reader hint (there is no drb:reader predicate any more). For the tile(s) a selection resolves to, RemapEngine.materialize(window):

1. expands the window to the enclosing physically-aligned region (drb/chunk/remap.py's aligned_window, clamped to the source's PhysicalChunkModel.array_shape);
2. reads that region once through the format's registered physical leaf reader (get_physical_reader(model.reader)) — the format add-on's only content-reading path; the core never imports a format library and calls no I/O itself;
3. caches it in a per-source BytesLRU region cache, so adjacent tiles sharing a native block only pay for it once;
4. crops the cached region down to the requested window.

A Chunk built with engine=None (no physical model on its source) is still valid for enumeration, select() and locator(), but raises DrbChunkError on get_impl/to_xarray — the add-on never silently falls back to reading the whole source. See docs/dev/architecture-layers.md and docs/dev/writing-a-physical-tiling-provider.md for how a format add-on supplies that leaf reader.

---

Locators and interop

chunk.locator()
# {'source': '/data/…/B04.tif', 'topic': 'http://…#my-image',
#  'chunk': 'b04', 'selection': {'window': {...}} | None}

from drb.chunk import to_kerchunk
to_kerchunk(chunk)      # {'version': 1, 'refs': {'1.0': [path, offset, length], …}}

to_kerchunk emits a kerchunk reference-spec v1 dict — a physical-address annex, orthogonal to the read path above. Because kerchunk addresses chunks by byte range, it only works for a ChunkManifest whose ChunkRef.byte_range is set; window-only (format-native) chunks, which is what every manifest shipped in the core produces, raise DrbChunkError. Populating byte_range is a manifest's choice (e.g. a content-driven add-on's ChunkManifest), independent of whether the chunk also has a RemapEngine for direct reads.

---

Exceptions

from drb.chunk import DrbChunkError, DrbSelectionError

• DrbChunkError — base error for the add-on (unknown chunk name, no physical model/engine for materialisation, unsupported get_impl, v1 deferrals, …). Subclasses drb.exceptions.core.DrbException.
• DrbSelectionError — unknown selection type or an out-of-bounds region. Subclasses DrbChunkError.

---

v1 limitations (intentional deferrals)

These are explicit, raise clear errors, and are tracked as follow-ups:

• Transformer: regular (RegularGrid) is the only built-in transformer; content-driven transformers (e.g. Sentinel-1 burst) come from add-ons.
• Multi-tile assembly: to_xarray() assembles a single tile; concat/mosaic across tiles is not yet implemented. get_impl(np.ndarray) likewise requires a single-tile selection.
• Descriptor-level drb:selection: a default selection in the descriptor is rejected at build time — apply selections explicitly via Chunk.select().
• Chunk.from_locator(): locator round-trip is deferred (locator() works).
• Label/coord & band/range resolution: SelSelection, BandSelection and RangeSelection are not resolved by RegularGrid in v1.
• Resampling transformers: a non-windowing transformer kind (e.g. a healpix resampling representation) would need its own mechanic distinct from RemapEngine; not designed or implemented yet.

---

Public API

from drb.chunk import (
    Chunk, ChunkAddon, ChunkArray, ChunkRef, ChunkManifest,
    Selection, parse_selection,
    Transformer, WindowingTransformer, RegularGrid, RemapEngine,
    TransformerProvider, BuildContext,
    register_transformer, get_transformer,
    PhysicalChunkModel, PhysicalTilingProvider,
    register_physical_provider, get_physical_provider,
    build_source_extractor,
    to_kerchunk, DrbChunkError, DrbSelectionError, __version__,
)

# ChunkAddon public methods (singleton via AddonManager().get_addon("chunk")):
#   .can_apply(topic)                                    -> bool
#   .available_chunks(topic)                             -> list[tuple[str, dict]]
#   .available_collections(topic)                        -> dict[str | None, list[str]]
#   .apply(node, *, chunk_name=…, transformer=…)          -> Chunk
#   .apply(node, *, collection=…, transformer=…)          -> list[Chunk]
#   .apply(node)                                          -> list[Chunk]  (all declared chunks)

The physical leaf-reader registry (register_physical_reader/ get_physical_reader) lives in drb.chunk.physical alongside PhysicalTilingProvider but is not re-exported at the top-level package; import it from drb.chunk.physical directly (as every format add-on does).

Development

python3 -m venv venv && source venv/bin/activate
pip install -e . -r requirements-test.txt
python3 -m unittest discover        # unittest suite under tests/

Three layers, two extension seams

drb-chunk is organised in three layers — transformer (user-facing representation), mechanics (RemapEngine, internal), physical (the source's native chunking) — see docs/dev/architecture-layers.md for the overview. Add-ons extend the two outer layers:

Extending: content-driven transformers

regular is the only transformer built into the core. An add-on can teach the engine a new one — geometry read from the product itself at apply-time — by shipping a drb.chunk.transformer entry point. See docs/dev/writing-a-transformer.md for the TransformerProvider contract and a worked example; drb-chunk-sentinel-1 (the burst transformer) is the reference implementation.

Extending: physical-tiling providers

drb-chunk discovers a source's native chunking (the unit of I/O) rather than taking it from the descriptor. An add-on teaches the engine a new format's native chunking by shipping a drb.chunk.physical entry point and attaching drb:physicalTiling "<token>" to the format's topic. See docs/dev/writing-a-physical-tiling-provider.md for the PhysicalTilingProvider contract and a worked example; drb-chunk-image (the gdal-blocks provider, for GDAL rasters) is the reference implementation.

Add-on naming convention

A chunk add-on's PyPI dist name is drb-chunk-<name> and its Python module is drb.addons.chunk.<name>. The product suffix is hyphenated in the dist name (drb-chunk-sentinel-1, drb-chunk-sentinel-2) while the module keeps a Python identifier (no hyphen: drb.addons.chunk.sentinel1) and the git repository keeps its directory name (sentinel-1). Two kinds of add-on live under add-ons/:

• product add-ons (drb-chunk-sentinel-1, drb-chunk-sentinel-2) — content-driven transformers / drb:chunk descriptors;
• format add-ons (drb-chunk-image, and future drb-chunk-netcdf / drb-chunk-zarr) — physical-tiling providers, one per implementation that exposes a native chunking.

(The chunk family deliberately hyphenates the product suffix in the dist name; other DRB families historically use the unhyphenated form, e.g. drb-image-sentinel1.)

License

LGPLv3 — see LICENCE.txt.