cng-datasets

A CLI toolkit for processing large geospatial datasets into cloud-native formats on Kubernetes.

What it does: Takes source geospatial data (Shapefiles, GeoPackages, FileGDB, GeoTIFFs) and produces:

• GeoParquet — columnar format for analytical queries (DuckDB, Polars)
• PMTiles — vector tiles for web map visualization
• H3 Hex Parquet — hexagonal grid indexed at configurable H3 resolution, hive-partitioned for fast spatial joins
• Cloud-Optimized GeoTIFF — for raster data, optimized for HTTP range requests

How it works: You run a single CLI command locally. It generates Kubernetes Job YAML files that orchestrate the entire pipeline on your cluster. You never process data on your local machine — the CLI just generates the jobs.

Installation

pip install cng-datasets

Or from source:

pip install -e "."

Usage

Generate a vector processing pipeline

cng-datasets workflow \
  --dataset cpad-2024 \
  --source-url https://example.com/cpad.gdb \
  --bucket public-cpad \
  --layer CPAD_SuperUnits \
  --h3-resolution 10 \
  --parent-resolutions "9,8,0" \
  --hex-memory 32Gi \
  --max-completions 200 \
  --max-parallelism 50

This generates YAML files for a 5-step pipeline:

1. setup-bucket — creates the S3 bucket with public-read policy
2. convert — reads source data, reprojects to EPSG:4326, writes GeoParquet
3. pmtiles — converts GeoParquet to PMTiles (parallel with step 4)
4. hex — computes H3 cell assignments in parallel pods
5. repartition — consolidates hex chunks into hive-partitioned layout

Multiple sources: Add --source-url multiple times to merge datasets:

cng-datasets workflow \
  --dataset merged-regions \
  --source-url https://example.com/region1.shp \
  --source-url https://example.com/region2.shp \
  --source-url https://example.com/region3.shp \
  --bucket my-bucket

Apply them:

kubectl apply -f <output-dir>/workflow-rbac.yaml
kubectl apply -f <output-dir>/configmap.yaml
kubectl apply -f <output-dir>/workflow.yaml

The workflow orchestrator runs steps sequentially, launching pmtiles and hex in parallel.

Generate a raster processing pipeline

cng-datasets raster-workflow \
  --dataset wetlands-cog \
  --source-url https://example.com/wetlands.tif \
  --bucket public-wetlands

Multi-layer sources

For GeoDatabase or GeoPackage files with multiple layers, run one workflow per layer. Use --layer to select each layer and --dataset with / for hierarchical S3 paths:

# Inspect layers
ogrinfo /vsicurl/https://example.com/data.gdb

# Process each layer separately
cng-datasets workflow --dataset mydata/layer-a --layer LayerA ...
cng-datasets workflow --dataset mydata/layer-b --layer LayerB ...

The / in --dataset creates nested S3 paths (e.g., mydata/layer-a.parquet) while using hyphens for k8s resource names.

CLI Reference

Commands marked "used inside k8s pods" are called by the generated jobs — you don't run them directly.

cng-datasets workflow options

--dataset NAME             Dataset name for S3 paths. Use / for hierarchy (e.g., "padus/fee").
--source-url URL           Public URL to source data.
--bucket BUCKET            Target S3 bucket name.
--layer LAYER              Layer name for multi-layer sources (GDB, GPKG).
--h3-resolution N          H3 resolution for hex tiling (default: 10).
--parent-resolutions STR   Comma-separated parent resolutions (default: "9,8,0").
--hex-memory SIZE          Memory per hex pod (default: 8Gi).
--max-completions N        Number of parallel chunks, max 200 (default: auto).
--max-parallelism N        Max concurrent pods (default: 50).
--id-column COL            ID column name (auto-detected if omitted).
--output-dir DIR           Directory for generated YAML files.
--intermediate-chunk-size  Rows per unnest batch (decrease if OOM).
--row-group-size N         Rows per parquet row group (default: 100000).

cng-datasets raster-workflow options

--dataset NAME             Dataset name for S3 paths. Use / for hierarchy.
--source-url URL           Public URL to a source raster. Repeat to mosaic tiles.
--bucket BUCKET            Target S3 bucket name.
--h3-resolution N          H3 resolution for hex tiling (default: 8).
--parent-resolutions STR   Comma-separated parent resolutions (default: "0").
--hex-resampling R         mean | sum | mode | fractions | max | min (default: mean).
--nodata VALS              NoData value(s) to exclude, comma-separated.
--hex-memory SIZE          Memory per hex pod (default: 32Gi).
--hex-cpu N                CPU request/limit per hex pod (default: 4).
--hex-workers N            Worker processes per hex pod (default: one per --hex-cpu).
--hex-chunk-size N         Cells per exact_extract call (default: 100000).
--h0-subset CELLS          The h0 base cells the source overlaps (default: all 122).
--chunk-resolution N       H3 resolution of one hex pod's unit of work (default: 0, one h0).
--max-hex-memory SIZE      Pick --chunk-resolution automatically to fit this budget.
--backend BACKEND          k8s (default) | armada | auto (armada past ~200 chunks).
--max-parallelism N        Max concurrent hex pods (default: 61).
--output-dir DIR           Directory for generated YAML files.

S3 Output Layout

bucket/
├── dataset.parquet              # GeoParquet
├── dataset.pmtiles              # PMTiles
├── dataset/
│   └── hex/
│       └── h0={cell}/data_0.parquet   # H3-indexed, hive-partitioned
├── README.md
└── stac-collection.json

Docker

The CLI and all dependencies are packaged in a Docker image used by the k8s jobs:

docker pull ghcr.io/boettiger-lab/datasets:latest

Troubleshooting

OOM on hex jobs (vector): Increase --hex-memory (e.g., 32Gi → 64Gi), increase --max-completions for smaller chunks, or decrease --intermediate-chunk-size.

Raster hex pod too big even at one worker: the pod's memory tracks the largest chunk's H3-cell count, and the default chunk is a whole h0 base cell (~282M cells at res 10). Pass --chunk-resolution 1 or 2 to raster-workflow to make the unit a descendant of an h0 instead — roughly 7x fewer cells per level — or --max-hex-memory 8Gi to have it chosen for you. The generated workflow gains a merge step that restores the usual h0={cell}/data_0.parquet layout, so the published layout is unchanged. Past ~200 chunks a single indexed Job exceeds the namespace pod guideline; --backend auto routes to Armada at that point. Sub-chunked pods read only their own window of the source COG rather than localizing the whole file (--window-reads), so total transfer stays proportional to the data instead of to the number of chunks.

OOM on hex jobs (raster): Lower --hex-workers before raising --hex-memory. Peak RSS is roughly --hex-workers × --hex-chunk-size × bytes per cell, and a memory request large enough to matter (past ~128Gi) makes the pod contend for scarce large-RAM nodes, turning a retryable OOM into an unschedulable one. Both knobs are always written into the manifest as CNG_HEX_WORKERS / CNG_HEX_CHUNK_SIZE, so a tuned pod survives regeneration.

Convert fails on curved geometries (MULTISURFACE): Handled automatically — the converter linearizes curved geometry types via ogr2ogr before processing.

Monitoring:

kubectl get jobs              # Pipeline status
kubectl logs job/<name>       # Job logs
kubectl get pods | grep OOM   # Check for memory issues

Development

pip install -e ".[dev]"
pytest tests/

License

Apache 2.0

Citation

If you use cng-datasets in your work, please cite it via its Zenodo archive. The DOI badge at the top of this README resolves to the latest release; each release also has its own version-specific DOI. Citation metadata (authors, title, funding) lives in .zenodo.json and is minted automatically by Zenodo on each GitHub release.

Boettiger, C., & Buhler, C. K. cng-datasets: A cloud-native toolkit for processing large geospatial datasets on Kubernetes. Zenodo. https://zenodo.org/badge/latestdoi/921944685

Once the first release is archived on Zenodo, replace the redirect above with the concrete versioned DOI it mints.