envlib

A distributed catalogue for environmental data

Documentation: https://envlib.github.io/envlib/

Source Code: https://github.com/envlib/envlib

Overview

envlib is a distributed database and catalogue for environmental datasets — gridded model output and station time series alike. Datasets are stored as cfdb files on S3-compatible object storage, hosted by whoever owns the data; envlib provides the shared layer on top: standardized metadata, controlled vocabularies, deterministic identifiers, and a catalogue you can query to discover and open any registered dataset.

Key features

• Distributed by design — the catalogue is an index, not a data silo: each dataset stays on its owner's storage, and one catalogue can span many owners.
• Standardized metadata — eleven identity fields (drawing on ODM2, CF conventions, and lessons from tethys) describe what every dataset is, validated against controlled vocabularies at creation time.
• Deterministic, permanent identifiers — a dataset's identity metadata hashes to stable ids (dataset_id, dataset_version_id, station_id), so the same data gets the same id everywhere, forever.
• Queryable catalogue — filter by any identity field, spatial extent (including across the antimeridian), and time range; browse what a catalogue holds with cat.variables, cat.owners, and friends.
• CF standard names derived for you — envlib curates the mapping from its variables to CF standard_names and applies it automatically at registration.
• cfdb storage — every dataset version is a cfdb file: chunked, compressed, partially readable, S3-syncable.

Installation

pip install envlib

Quick example

import envlib

# The public envlib catalogue — zero config, no credentials
# (until it's hosted: Catalogue(remotes=['https://.../catalogue.rcg']))
cat = envlib.Catalogue()

# What's in it?
cat.variables       # ['precipitation', 'streamflow', 'temperature']
cat.owners          # ['ecan', 'ecmwf', 'niwa']

# Find the latest version of each matching dataset
results = cat.query(
    variable='temperature',
    feature='atmosphere',
    bbox=[166, -47, 179, -34],
    start_date='2020-01-01',
)

# Open one as a cfdb dataset and read it
ds = results[0].open()
temp = ds['temperature']

Producing data is the same library in the other direction: build a cfdb file, attach envlib.Metadata, and cat.publish(...) it — see the documentation for the full guide.

Changes between releases are tracked in the changelog.

Development

Setup environment

We use UV to manage the development environment and production build.

uv sync

Run unit tests

You can run all the tests with:

uv run pytest

Format the code

Execute the following commands to apply linting and check typing:

uv run ruff check .
uv run black --check --diff .
uv run mypy --install-types --non-interactive envlib

To auto-format:

uv run black .
uv run ruff check --fix .

License

This project is licensed under the terms of the Apache Software License 2.0.