usdata
Reproducible acquisition of U.S. public scientific data. One Python SDK and CLI discovers curated datasets from agencies such as NOAA, USGS, and EPA, fetches their files, and keeps a record of every input: a manifest names them, a lockfile pins them by checksum, and the record of what was fetched is what a methods section cites. Analysis stays in pandas and xarray; usdata only acquires. If you need every product one agency publishes, that agency's own library is the better tool; usdata is for pinning inputs across sources and proving later that they have not changed.
pip install "usdata[pandas]"
usdata search precipitation --location Oklahoma
usdata fetch noaa:ghcn-daily -p stations=USW00013967 --start 2024-05-06 --end 2024-05-07
Save the same station query as dataset.yaml in your working directory:
name: first-station
sources:
- dataset: noaa:ghcn-daily
start: 2024-05-06
end: 2024-05-07
params:
stations: USW00013967
Then fetch and pin its inputs, and print their citation:
usdata pull dataset.yaml && usdata cite dataset.yaml
The first pull writes dataset.lock.json; later pulls restore its pinned assets.
From Python, using the same dataset.yaml:
from usdata import build_query, fetch, get, pull
query = build_query(start="2024-05-06", end="2024-05-07", stations="USW00013967")
items = fetch(get("noaa:ghcn-daily"), query)
frame = items[0].open() # pandas DataFrame, units and provenance in frame.attrs
result = pull("dataset.yaml") # every manifest source, pinned in dataset.lock.json
print(result.fetched[0].provenance.checksum) # what usdata cite reports
Alpha. Releases are tested and lockfiles are stable, but the Python API and CLI can change between minor versions; the changelog marks breaking changes.
Where to look
| usdata.dev | What it is for: the dataset browser and worked examples with saved results |
| docs.usdata.dev | How to use it: install, getting started, guides, dataset notes, and reference |
| Severe-weather case study | One tornado, six sources, one manifest and lockfile, ending in a citation |
Twenty-seven datasets are available today and nineteen more are planned, grouped by agency and product family in the catalog.
How this compares
usdata resolves a place, a time window, and a few parameters to the exact files that satisfy them, then pins those files so the same inputs can be restored and verified later. Neighbouring tools each do part of that.
| Tool | What it does | What usdata adds |
|---|---|---|
| pooch | Downloads and verifies files from a registry of names, hashes, and a base URL you write | Discovery: the query resolves to the URLs, and the lockfile is written, not authored |
| intake | Describes datasets in catalogs and opens them through drivers | Checksums, lockfiles, and provenance; usdata stops at handing a file to pandas or xarray |
| DVC | Versions data files and can track one URL by its ETag or hash | Knowing which URLs a scientific question needs, across agencies |
| Herbie | Deep access to model output, including subsetting GRIB2 through index files | One interface across observations, radar, satellite, and models, with a lockfile; usdata's index-file fetch reproduces Herbie's for that reason |
| dataretrieval | Full coverage of USGS water data | Cross-agency pinning; use dataretrieval for USGS breadth |
Contributing
Setup, commands, checks, and the release procedure are in CONTRIBUTING.md. The project record lives beside it: architecture, testing, decisions, roadmap, and versioning. The main way the project grows is a new dataset, following adding a dataset.
License
1 maintainer
Release files for usdata 0.29.0
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| usdata-0.29.0.tar.gz | 289.9 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| usdata-0.29.0-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 624.0 kB
Release files / usdata-0.29.0.tar.gz
| Download URL | usdata-0.29.0.tar.gz |
|---|---|
| Size | 289.9 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
6184770c041c9e7df6b3ca6ba05d38064b767bfa29b8b622fbd99f01311bdec8
|
|
BLAKE2b-256 checksum How to use checksums |
03ef3ef92e292e0ada59a03e54a9aae97c875ee5ad110ccef44e8db84d718598
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.12.19 {"installer":{"name":"uv","version":"0.12.19","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
Release files / usdata-0.29.0-py3-none-any.whl
| Download URL | usdata-0.29.0-py3-none-any.whl |
|---|---|
| Size | 334.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
82c8580ecafe66c33d413b51597be894ed8ed672a1968e3b0ac4214937cb4d36
|
|
BLAKE2b-256 checksum How to use checksums |
5f51ce66434e8e455928ac2a02eab4386923b19697c13c9480e9896a059a075a
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
Yes |
| Uploaded via |
uv/0.12.19 {"installer":{"name":"uv","version":"0.12.19","subcommand":["publish"]},"python":null,"implementation":{"name":null,"version":null},"distro":{"name":"Ubuntu","version":"24.04","id":"noble","libc":null},"system":{"name":null,"release":null},"cpu":null,"openssl_version":null,"setuptools_version":null,"rustc_version":null,"ci":true}
|
