usdata

Unified Python SDK and CLI for discovering, fetching, and tracking the provenance of U.S. public scientific data (NOAA, USGS, NASA, and more).

Status: pre-alpha. noaa:ghcn-daily and noaa:nexrad-level2 fetch real data with provenance; other registry entries are stubs. See docs/roadmap.md.

Providers

Available datasets are in code, stubs in italics; planned ones are counted. Each provider page has access notes and full dataset details; docs/roadmap.md lists datasets by target version.

Install

pip install usdata        # or: uv add usdata

Usage

from usdata import build_query, get, search
from usdata.fetch import fetch

for r in search("precipitation", location="Oklahoma"):
    print(r.dataset.id, r.dataset.title)

ds = get("noaa:ghcn-daily")
query = build_query(
    lat=35.39,
    lon=-97.60,
    radius_km=15,
    start="2024-05-06",
    end="2024-05-07",
    variables=["PRCP", "TMAX"],
)
for item in fetch(ds, query):
    print(item.path, item.provenance.checksum)

usdata search "tornado radar" --state OK
usdata info noaa:ghcn-daily
usdata fetch noaa:ghcn-daily --lat 35.39 --lon -97.60 --radius-km 15 \
    --start 2024-05-06 --end 2024-05-07 --vars PRCP,TMAX
usdata fetch noaa:ghcn-daily -p stations=USW00013967 --start 2024-01-01 --end 2024-12-31
usdata fetch noaa:nexrad-level2 --lat 35.47 --lon -97.52 \
    --start 2024-05-06T20:00 --end 2024-05-06T23:00        # nearest radar (KTLX)
usdata fetch noaa:nexrad-level2 -p site=KTLX --start 2024-05-06T20:00 --end 2024-05-06T20:30 --dry-run
usdata pull dataset.yaml            # resolve, fetch, write dataset.lock.json
usdata verify dataset.yaml          # exit 1 if any cached input drifted

Fetched files land in ~/.cache/usdata/<provider>/<dataset>/ (override with USDATA_CACHE_DIR or --cache-dir), each with a .provenance.json sidecar recording source URL, retrieval time, checksum, size, and license.

Manifest and source fields are validated strictly; unknown fields are errors. Provider-specific options belong under params.

A manifest declares every input a project needs. pull resolves each source, fetches it, and writes dataset.lock.json pinning every asset with its checksum and provenance. A second pull restores exactly what the lockfile pins without re-querying upstream, so the inputs stay reproducible even if the source changes. verify re-hashes the cached files against the lockfile. Editing the manifest after locking requires pull --force to re-resolve.

name: tornado-environment
sources:
  - dataset: noaa:nexrad-level2
    location: oklahoma
    start: 2024-05-06
    end: 2024-05-07
  - dataset: noaa:ghcn-daily
    location: oklahoma
    start: 2024-05-01
    end: 2024-05-31

Development

Requires uv and just.

git clone https://github.com/jakeryderv/usdata && cd usdata
just setup     # install toolchain and dependencies
just test      # unit tests
just check     # format, lint, typecheck, tests (what CI runs)
just run search radar

Integration tests that hit live services run with just test-integration.

Releases: just release minor opens a version-bump PR; merging it publishes to PyPI and creates the tag and GitHub release. See docs/versioning.md.

See docs/providers/ for per-provider access notes, docs/architecture.md for how the pieces fit, docs/adr/ for why, and CONTRIBUTING.md to add a dataset.

License

Apache-2.0