CRC SDK

CRC SDK is the higher-level Python interface for Climate Risk Commons data access, storage providers, geometry utilities, and analytical workflows. Numerical distributions, curve fitting, impact transforms, and risk metrics are provided by the versioned crc-framework dependency.

Development

uv is the recommended tool for managing the development environment:

uv sync --all-extras

uv run pytest
uv run mypy
uv run ruff check .

Or simply:

python -m venv .venv
.venv/bin/python -m pip install -e ".[zarr,raster,geometry,test]"
.venv/bin/python -m pytest
.venv/bin/python -m mypy
.venv/bin/python -m ruff check .

Dependencies

DuckDB, Arrow (pyarrow), psutil (resource detection), and remote-storage transport (fsspec, s3fs, gcsfs) are baseline dependencies — every connector and workflow in this SDK is built on that stack, so gating it behind an extra would just move the same install onto every real caller.

Everything else is a specific data-format adapter or a pure-geometry dependency, opted into only by the callers that need it:

DuckDB's community duckdb_zarr extension can scan ordinary remote Zarr v2/v3 stores, but it does not open Icechunk's versioned repository/session model. USDA CDL therefore uses the official Icechunk client for version resolution and chunk reads, then exposes bounded Arrow batches to DuckDB; it never materializes the full raster or a full AOI in memory.

Every function that needs an extra-gated dependency imports it lazily and raises a clear ImportError naming the extra to install if it's missing — importing crc_sdk (or any of its subpackages) itself never requires more than the baseline dependencies.

OS-level dependency (not a pip extra): tippecanoe and tile-join (https://github.com/felt/tippecanoe) must be present on PATH for crc_sdk.geometry.pmtiles — they're assumed to already be installed on the runtime image, not pip install-able, so there's no extra for them. Verify availability with require_tippecanoe()/require_tile_join(), which raise a friendly, actionable error (with install instructions) if either is missing.

Package boundaries

• crc_sdk.core, crc_sdk.fitting, and crc_sdk.impacts expose the stable public API of crc_framework.
• crc_sdk.connectors handles external formats and query engines: DuckDB connection helpers (DuckDBConnection, RuntimeResources, streaming Parquet writes), OS-Climate Zarr ingest (zarr extra), and GeoTIFF/COG ingest (GeoTiffRaster, raster extra) — the latter streams directly from local paths or gs:///s3:///http(s):// URIs via GDAL's own range-request support, with no local download by default. DuckDBRelationSource, ArrowBatchSource, and DuckDBPipeline form the common lazy process seam: native SQL/Parquet adapters return relations, while chunk stores yield bounded Arrow batches into the same immutable filter/project/aggregate/write pipeline. AgriculturalLayer.usda_cdl() uses that Arrow seam for the versioned Icechunk v2 CDL store; AgriculturalLayer.ftw_fields() stays native in DuckDB over remote GeoParquet, applying bbox pruning before exact spatial filtering.
• crc_sdk.providers describes storage and dataset discovery.
• crc_sdk.geometry contains geometry conversion, DuckDB-native H3 polyfill (H3Indexer), Arrow batch polyfill (polyfill_wkb, geometry extra for h3ronpy), raster-to-H3 sampling primitives (pixel_grid_resolution, sample_grid_to_h3), exploded coverage writers (write_exploded_coverage), optional nested lookup derivation (LookupCatalog, write_lookup_contract, write_partitioned_lookup), and PMTiles generation (crc_sdk.geometry.pmtiles — also reachable flattened as crc_sdk.geometry.PMTilesBuild, etc.): PMTilesBuild streams a GeoParquet source (a single file, or a Hive-partitioned dataset glob) into one .pmtiles archive in one tiling pass, building the GeoParquet -> GeoJSON bridge itself in DuckDB spatial-extension SQL (ST_AsGeoJSON/ST_ReducePrecision/ST_Transform) rather than shelling out to an external converter, streamed via the same Arrow-batched-reader pattern used elsewhere in this SDK. tippecanoe_threads/duckdb_threads default to every detected core (no conservative per-thread cap, unlike DuckDB's own GEOS-throttled default) since tippecanoe's tile-building has no documented per-thread memory ceiling. A pre-flight budget check raises a clear, actionable error if a source is estimated to exceed available scratch disk, rather than silently degrading into a slower multi-batch fallback — provisioning more disk or narrowing the run's scope is left to the caller.
• crc_sdk.schema defines columnar data contracts.
• crc_sdk.types contains SDK-owned Pydantic configuration and metadata.
• crc_sdk.workflows coordinates data access and computation.

DuckDB resource limits are detected when requested (RuntimeResources.detect / DuckDBConnection.for_analytics) and relayed through the connection config mapping. Thread count is min(cpus, usable_RAM / GiB_per_thread) with usable RAM ≈ 60% of detected memory and a default of ~2.5 GiB/thread (GEOS spatial work often slows when over-threaded). memory_limit and max_temp_directory_size remain hard process caps. Override with CRC_DUCKDB_THREADS, CRC_DUCKDB_MEMORY, and/or CRC_DUCKDB_BYTES_PER_THREAD_GIB, or pass an explicit config dict. Set CRC_DUCKDB_PROFILE=1 to enable detailed query profiling around enrich/coverage stages in h3geo.

Constructors with no natural caller-supplied directory of their own (OSClimateProvider, ZarrRaster, H3Indexer) build a resource-tuned connection by default — via DuckDBConnection.for_analytics — instead of a bare, untuned one, so this scales out of the box with no configuration. Passing an explicit connection/con always wins and skips this entirely. Otherwise the spill/temp directory defaults to a stable location under the system temp directory (default_work_dir(), not a fresh one per call), and can be set per-call via each constructor's own work_dir parameter, or globally via CRC_DUCKDB_WORK_DIR.

Private/authenticated remote sources (a non-public GCS/S3 bucket) are configured the same idiomatic-DuckDB way as everything else here: raw CREATE OR REPLACE SECRET SQL, passed as setup_sql=(...) to DuckDBConnection/DuckDBConnection.for_analytics to have it run automatically on .connect(), right after extensions load. There is deliberately no secret-builder type in the SDK — DuckDB's own secret DDL (https://duckdb.org/docs/configuration/secrets_manager) is already the documented interface, and a caller's own connection-setup module is a more natural home for its specific credentials than a generic wrapper trying to track every provider/type DuckDB supports. sql_quote/sql_identifier are exported for safely building that SQL; the one DuckDB quirk worth knowing is that PROVIDER is a bare keyword (config, credential_chain, ...), not a quoted string literal, unlike every other secret option:

import os
from crc_sdk.connectors.duckdb import DuckDBConnection, sql_quote

setup_sql = []
key_id, secret = os.getenv("GCS_ACCESS_KEY"), os.getenv("GCS_ACCESS_SECRET")
if key_id and secret:
    setup_sql.append(
        f"CREATE OR REPLACE SECRET gcs (TYPE GCS, KEY_ID {sql_quote(key_id)}, "
        f"SECRET {sql_quote(secret)})"
    )
con = DuckDBConnection.for_analytics(work_dir, setup_sql=setup_sql).connect()

Agricultural layers

Agricultural requests are immutable and bounded before they can scan. Builder calls perform no network I/O; relation(), to_arrow_reader(), and write_parquet() are execution points:

from crc_sdk.workflows import AgriculturalLayer

crop_mix = (
    AgriculturalLayer.usda_cdl()
    .resolution("30m")
    .for_area((-93.46, 42.14, -93.45, 42.15))
    .years(2025)
    .classes([1, 5])  # corn and soybeans
    .scan()
    .pipeline()
    .aggregate("count(*) AS sampled_pixels", groups="year, crop_code, crop_name")
)
for batch in crop_mix.to_arrow_reader():
    process(batch)

For global predicted field units, replace the source while keeping the same process surface:

fields = (
    AgriculturalLayer.ftw_fields()
    .in_country("FR")
    .for_area((2.0, 47.5, 3.0, 48.5))
    .years(2024)
    .confidence_at_least(80)
    .scan()
    .pipeline()
)
fields.write_parquet("outputs/france-fields.parquet")

FTW fields are remote-sensing units, not cadastral parcels or evidence of ownership. CDL crop pixels are land-cover observations, not acreage, yield, or financial exposure.

Canonical hazard datasets

The SDK internalizes fitted hazards as one versioned Arrow/Parquet contract. Rows contain a canonical unsigned H3 cell_index, stable source_id, optional source WKB, scenario dimensions, and the parameters needed to reconstruct a crc_framework.FittedDistribution, HurdleDistribution, PointMassDistribution, or compact TabulatedDistribution. curve_shape is nullable because Gumbel families do not use a shape parameter; atom probability and location are populated for hurdle and point-mass rows. Schema 1.1 added curve_kind="point_mass" for a distribution that is constant across the complete source probability support. It uses the same physical columns, with curve_type="point_mass", zero scale, and probability one at curve_location; downstream quantile calls remain identical to fitted and hurdle curves.

Schema 1.2 adds two nullable list columns, curve_probabilities and curve_values. They are populated only for curve_kind="tabulated", with curve_type="linear_probability"; scalar curve parameters are null. It also supports curve_kind="no_data", whose curve_type is an explicit scientific reason code and whose parameter fields are all null. Batch quantile evaluation returns nulls for those rows. This tagged-union layout avoids redundant status, fit-stage, interpolation, and reason columns: dataset metadata and run manifests carry ordered-family and aggregate treatment provenance once.

The logical row key is (hazard_name, horizon, pathway, cell_index, source_id). cell_index is the spatial join key, not a globally unique identifier. Canonical files are sorted by that row key for predicate pruning and merge joins by default.

write_hazard_stream(..., ordered=True) retains that default contract. DuckDB materializes the canonical stream, rejects duplicate keys, globally sorts, and may spill to the configured work directory after reaching its memory limit. This keeps engine memory predictable rather than batch-constant: reserve additional process headroom for Arrow/Python buffers and the final Parquet write. Ordered output generally compresses better and can improve scans that benefit from physical key clustering.

ordered=False is an explicit low-memory alternative for unusually large partitions or tighter containers. It appends validated Arrow batches directly to a local Parquet staging file, scans only projected row-key columns for duplicates, and atomically publishes after validation. Canonical schema, metadata, uniqueness, and downstream curve/percentile APIs are identical, but physical rows retain input order rather than the global canonical sort.

On one 2,148,497-row schema-1.2 sample, direct streaming used 656 MB peak RSS and produced a 45 MB file in 16.03 seconds; ordered writing used 1.69 GB and produced a 35 MB file in 16.38 seconds. A 300,000-row sample used 282 versus 525 MB, produced 6.3 versus 4.9 MB, and took 3.01 versus 2.78 seconds. These measure the persistence pass only on one machine: storage reduction was consistent, while the small timing difference changed direction. Treat them as tradeoff evidence, not universal throughput guarantees.

Dataset-wide facts are stored once as a complete JSON payload under the crc.hazard.metadata Parquet key: schema version, one uncompacted H3 resolution, non-exceedance probability convention, source probability support, value unit and semantics, WKB CRS, producer, source provenance, curve-fit policy, and creation version.

Each dataset is one self-describing Parquet file, expanded by H3 cell for spatial joins. The caller chooses its full destination path and filename. Writes use DuckDB, and an optional configured DuckDB connection allows the same API to use its local or cloud filesystems, extensions, secrets, and settings. Source knots and fit diagnostics are transient ingest inputs, not a second persisted data contract. Values at source return periods are therefore fitted curve evaluations rather than guaranteed bit-for-bit reproductions of source pixels.

External connectors remain source-format readers. Ingest adapters perform the explicit conversion:

external raster/table -> source curves and geometry -> selected family fit
  -> conservative intersecting H3 cells -> canonical Arrow -> Parquet

Boundary candidate generation uses H3 overlap coverage, not center polyfill. This makes the integer join a conservative superset before an exact ST_Contains(source_geometry, asset_point) refinement. Resolution estimates report measured coverage error and expanded row count, while ingest policy selects and records the dataset resolution.

OS-Climate return-period rasters can be canonicalized with OSClimateIngestPolicy and canonicalize_os_climate. The caller must choose the distribution family and, for zero-heavy hazards, provide an explicit HurdleFitPolicy; the SDK does not infer an exact point mass from sparse knots. Plain curves use fit_quantiles, while hurdle curves use fit_hurdle_quantiles. LocalProvider queries persisted hazard rows through HazardQuery.

Ingesting JRC flood maps

JRC flood acquisition is available as an immutable, lazy workflow. GLOFAS 2.1.2 uses JRC's tiled global layout; EFAS 3.1.1 uses nine continental return-period rasters. The dataset descriptions encode that difference so an AOI workflow does not expose tiles or filenames:

from crc_sdk.workflows import HazardDataset, JRCFloodPolicy

plan = (
    HazardDataset.efas(version="latest")
    .for_area((7.75, 49.75, 8.45, 50.25))
    .cache("cache/efas", mode="reuse")
    .source_periods("all")
    .canonicalize(
        policy=JRCFloodPolicy.curated(h3_resolution=10),
    )
)

print(plan.explain())
hazard = plan.materialize("hazards/efas-rhine.parquet")

Builder methods and explain() do not resolve releases or fetch rasters. materialize() resolves latest once, records the pinned version in the cache manifest and canonical provenance, caches AOI crops, fits the canonical curves, and returns an ordinary HazardDataset. Use prefetch() to populate the source cache separately, then switch the same plan to mode="offline". refresh explicitly re-resolves the release and replaces cached crops; stream reads remotely without a persistent source cache.

Source periods choose the rasters used for fitting and default to every period in the resolved release. They are distinct from evaluation periods:

result = (
    plan.for_assets(assets)
    .select(hazard_names=["RiverineInundation"], pathways=["historical"])
    .return_periods([50, 100, 250, 500])
    .write_parquet("outputs/flood-depth.parquet")
)

The compact chain lazily materializes a deterministic canonical file inside the configured cache, then delegates to the same local portfolio evaluator used by HazardDataset.local(...).

Canonical metadata records the source return-period support. Portfolio evaluation warns when a requested period falls outside it: with EFAS source rasters from RP10 through RP500, RP250 is interpolation while RP1000 is extrapolation.

Ingesting JRC/EDO drought data

EDO Soil Moisture Index data uses the same lazy workflow, with complete years reduced to compact annual-minimum AOI cache objects before fitting:

from crc_sdk.workflows import EDODroughtPolicy, HazardDataset

plan = (
    HazardDataset.smi(version="latest")
    .for_area((9.5, 50.5, 10.5, 51.5))
    .years("all_complete")
    .cache("cache/edo-smi", mode="reuse")
    .canonicalize(policy=EDODroughtPolicy.curated(h3_resolution=6))
)

hazard = plan.materialize("hazards/edo-smi.parquet")

The curated policy uses the lower return-period tail and requires at least 20 complete years. Metadata stores both that tail and the Gringorten support of the selected annual record. Evaluation therefore selects the correct lower tail automatically and warns when a requested period is extrapolated. Cache manifests pin the resolved EDO version, years, bounds, source URLs, local objects, and checksums; prefetch() followed by mode="offline" avoids later network access.

Evaluating asset portfolios at return periods

Canonical curve parameters can be evaluated for a portfolio without returning to the external source format or refitting the data. The workflow joins every asset to its canonical curve and writes one row per asset, hazard, horizon, and pathway, with one value column per requested return period:

import pyarrow as pa

from crc_sdk.workflows import HazardDataset

assets = pa.table(
    {
        "asset_id": ["warehouse-a", "warehouse-b"],
        "longitude": [6.9603, 7.5010],
        "latitude": [50.9375, 51.0030],
        "sector": ["logistics", "manufacturing"],
    }
)

result = (
    HazardDataset.local("flood.parquet")
    .for_assets(assets)
    .select(horizons=[2050], pathways=["ssp585"])
    .return_periods([25, 50, 100, 250, 500, 1000])
    .write_parquet("portfolio-flood.parquet")
)

The resulting value columns are value_rp25, value_rp50, value_rp100, value_rp250, value_rp500, and value_rp1000. For upper-tail hazards, each return period RP is evaluated at non-exceedance probability 1 - 1/RP. Value unit, value semantics, and the complete return-period/probability/column mapping are stored under crc.hazard.evaluation in Parquet metadata.

An impact function can replace the sampled hazard values with event-aligned impact values before writing:

import numpy as np

impact_result = (
    HazardDataset.local("flood.parquet")
    .for_assets(assets)
    .return_periods([25, 100, 250])
    .impact(
        lambda depth: np.clip(depth / 2.0, 0.0, 1.0),
        name="depth_damage_ratio",
        value_unit="fraction",
        value_semantics="damage ratio",
    )
    .write_parquet(
        "portfolio-impact.parquet",
        execution=ExecutionOptions(max_workers=1),
    )
)

The SDK first samples each hazard return period and then calls impact.evaluate(...) on that row's value vector. Therefore value_rp100 = impact(hazard_rp100): the return period continues to identify the source hazard event. This differs intentionally from transforming a full distribution and then taking an impact quantile, which can reorder decreasing or non-monotonic impacts. Use the distribution interface in crc-framework for that risk-analysis interpretation.

Built-in and registry-backed framework impacts use the same fluent method:

from crc_sdk.impacts import PiecewiseLinearImpact, impacts
from crc_sdk.workflows import ImpactContextColumns

damage_curve = PiecewiseLinearImpact(
    exposure=[0.0, 0.2, 1.0, 2.0],
    impact=[0.0, 0.0, 0.25, 1.0],
)

request = request.impact(
    damage_curve,
    name="flood_damage_ratio",
    value_unit="fraction",
    value_semantics="damage ratio",
)

registry_request = request.impact(
    impacts.for_factor("inundation"),
    context=ImpactContextColumns(
        country="country",
        continent="continent",
        building_type="building_type",
        historic_mean="historic_mean",
    ),
    name="inundation_impact",
    value_unit="fraction",
    value_semantics="damage ratio",
)

The generated H3 cell_index is always supplied to the framework impact context. Configured context columns are read from each asset, including when they are not retained as output passthrough columns. Stored registry context provides fallback values for fields without an asset value. Impact metadata records the event-aligned interpretation, source hazard units and semantics, output units and semantics, function name/type, and context-column mapping.

Framework impact objects and top-level Python callables can run in the existing process pool. Lambdas and closures are not picklable, so they run serially when the worker count is implicit; explicitly requesting more than one worker for one raises an error.

Point assets are converted to the H3 resolution recorded by the canonical dataset. The H3 join is refined with ST_Covers(source_geometry, asset_point) when source WKB is present; rows without WKB retain cell-level precision. source_id and spatial_match (exact_geometry or h3_cell) remain in the output. Multiple source curves for one asset/hazard/horizon/pathway raise instead of being silently selected or aggregated, and missing asset/scenario matches raise rather than being dropped from the output.

When assets already contain canonical H3 indexes, use cell_index_column="cell_index" instead of longitude/latitude columns. This avoids point conversion and exact source-geometry refinement:

(
    HazardDataset.local("flood.parquet")
    .for_assets(assets_with_cells)
    .return_periods([25, 50, 100, 250, 500, 1000])
    .write_parquet("portfolio-flood.parquet")
)

Arrow tables can be registered directly as shown above. A Path reads an asset Parquet file, while a string is treated as caller-supplied DuckDB SQL. Output evaluation is streamed in bounded Arrow batches to compressed Parquet. For already selected canonical rows, distribution_from_hazard_row remains available as the low-level curve reconstruction utility.

The common column names asset_id, longitude/latitude, and cell_index are inferred. Use AssetPortfolio, PointColumns, or CellColumn only for a nonstandard asset schema. Worker, batch, and connection controls are grouped under ExecutionOptions on write_parquet, keeping execution tuning out of the normal workflow.

License

CRC SDK is licensed under the GNU Affero General Public License, version 3 or later.