arraylake-datahub

DataHub ingestion source for Arraylake / Icechunk datasets.

The plugin crawls the Arraylake catalog over HTTPS and emits one DataHub Dataset per xarray-compatible Zarr group in your repos. Access control, credential vending, and querying stay in Arraylake — DataHub becomes the discovery and metadata-search surface, with externalUrl links back into the Arraylake web app for the actual data.

📖 Full documentation: Arraylake docs › Integrations › Catalogs › DataHub

What you get in DataHub

For every xarray-compatible group, a Dataset named <org>/<repo>/<group_path> with:

• Schema — one field per Zarr array. Coordinates and data variables are distinguished via a classification flag in each field's jsonProps, alongside shape, chunk_shape, dimension_names, codecs, and the full CF attribute bag (GRIB_* keys are filtered as noise).
• Description — the group's CF title + summary when present, otherwise the repo description.
• externalUrl — direct link into the Arraylake page for that group.
• customProperties spread:
  • Arraylake metadata: provider, product_type, spatial/temporal coverage, spatial_resolution, update_freq, etc.
  • Storage: bucket platform/name/region, computed storage_uri.
  • CF group attributes: license, institution, creator/publisher, time and geospatial coverage, references, history.
  • Marketplace subscription details if the repo is from a listing.

Repos with no xarray-compatible groups still emit one Dataset (repo landing only) so every catalog entry is discoverable.

Orphan repos — catalog entries whose underlying Icechunk storage no longer exists — are tagged with arraylake_storage_status=orphan.

Install

pip install arraylake-datahub 'acryl-datahub[datahub-rest]'

One-time platform registration

Register the earthmover custom data platform in your DataHub instance once:

datahub put platform \
  --name earthmover \
  --display_name "Earthmover" \
  --logo https://app.earthmover.io/icon.svg

Run

Save the following as recipe.yml:

source:
  type: earthmover
  config:
    # token: ${ARRAYLAKE_TOKEN}      # default: read from env
    # api_url: https://api.earthmover.io
    orgs:                            # omit to crawl every org the token sees
      - earthmover-public
    repo_pattern:
      allow: [".*"]
      # deny: [".*-archive$"]
    env: PROD                        # DataHub fabric

sink:
  type: datahub-rest
  config:
    server: http://localhost:8080
    token: ${DATAHUB_GMS_TOKEN}

Then run:

export ARRAYLAKE_TOKEN=ema_xxxxxxxxxxxx
export DATAHUB_GMS_TOKEN=...

datahub ingest -c recipe.yml --preview   # dry run
datahub ingest -c recipe.yml             # for real

The most useful knobs are orgs (allowlist) and repo_pattern (regex allow/deny). Full config below.

Config

Required Arraylake API access

The token needs read access to:

• GET /user/orgs
• GET /orgs/{org}/repos/paginated
• GET /repos/{org}/{repo}
• GET /repos/icechunk/{org}/{repo}/dataset-node

Verification

After a successful ingest, in DataHub's UI you should see:

• The Earthmover platform with logo.
• One Dataset per xarray-compatible Zarr group, named <org>/<repo>/<group>.
• A Schema panel listing every coordinate and data variable with units and CF descriptions where available.
• A clickable "View in Source" link that lands on the Arraylake page for that group, where authentication and querying happen.

Tested against acryl-datahub 0.15.x on Python 3.10–3.13.

Arraylake Support

Email — support@earthmover.io

Email us with any questions, bug reports, or feature requests.

License

Apache-2.0