Find and load data from the Brightband AI-ready mirror of the NOAA NASA Joint Archive (NNJA) of Observations for Earth System Reanalysis

Navigation

Verified details

These details have been verified by PyPI
Project links
GitHub Statistics
Maintainers
Avatar for darothen from gravatar.com darothen Avatar for hansmohrmann from gravatar.com hansmohrmann

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License (MIT License)
  • Tags NNJA , artificial intelligence , earth system reanalysis , data catalog , dataset , earth observation
  • Requires: Python >=3.10
  • Provides-Extra: interactive , dask , polars , complete
Classifiers
Report project as malware

Project description

GitHub Actions - CI ReadTheDocs - Status PyPI - Latest zenodo

nnja-ai: multi-modal, AI-ready weather observations

This is the companion Python SDK to the Brightband AI-ready reprocessing of the NOAA NASA Joint Archive (NNJA). It is meant to serve as a helpful interface between a user and the underlying NNJA datasets (which currently consist of parquet files on GCS).

The V1 release of the NNJA-AI dataset and SDK represents a major increment in availability of NNJA data, with ~50 TiB of observations made available in parquet form along with a data catalog and code examples in this SDK.

Background

The NNJA archive project is a curated archive of Earth system data from 1979 to present. This data represents a rich trove of observational data for use in AI weather modelling, however the archival format in which the data is originally available (BUFR) is cumbersome to work with. In partnership with NOAA, Brightband is processing that data to make it more accessible to the community.

Data

NNJA datasets are organized by sensor/source (e.g. all-sky radiances from the GOES ABI). The list of all NNJA datasets can be found on the NNJA project page, while the subset that is currently found in the NNJA-AI archive can be found here or by exploring the data catalog (this will be be expanding rapidly).

Getting Started

To install this package directly from the GitHub repository, you can use the following pip command:

pip install git+https://github.com/brightbandtech/nnja-ai.git

You can find an example notebook here showing the basics of opening the data catalog, finding a dataset, subsetting, and finally loading the data to pandas. Though to get started, you can open the data catalog like so:

from nnja_ai import DataCatalog
catalog = DataCatalog()
print("datasets in catalog:", catalog.list_datasets())

datasets in catalog:

['amsua-1bamua-NC021023',
 'atms-atms-NC021203',
 'mhs-1bmhs-NC021027',
 'cris-crisf4-NC021206',
 ...]

How to Cite

If you use this library or the Brightband reprocessed NNJA data, please cite it using the following DOI:

DOI

Additionally, please follow the citation guidance on the NNJA project page.

The NNJA-AI data is distributed with the same license as the original NNJA data, CC BY 4.0.

Project details

Verified details

These details have been verified by PyPI
Project links
GitHub Statistics
Maintainers
Avatar for darothen from gravatar.com darothen Avatar for hansmohrmann from gravatar.com hansmohrmann

Unverified details

These details have not been verified by PyPI
Project links
Meta
  • License: MIT License (MIT License)
  • Tags NNJA , artificial intelligence , earth system reanalysis , data catalog , dataset , earth observation
  • Requires: Python >=3.10
  • Provides-Extra: interactive , dask , polars , complete
Classifiers

Release history Release notifications | RSS feed

This version

1.0.0

0.2.1

0.2.0

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

nnja_ai-1.0.0.tar.gz (18.6 kB view details)

Uploaded Source

Built Distribution

If you're not sure about the file name format, learn more about wheel file names.

nnja_ai-1.0.0-py3-none-any.whl (17.8 kB view details)

Uploaded Python 3

File details

Details for the file nnja_ai-1.0.0.tar.gz.

File metadata

  • Download URL: nnja_ai-1.0.0.tar.gz
  • Upload date:
  • Size: 18.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.12.9

File hashes

Hashes for nnja_ai-1.0.0.tar.gz
Algorithm Hash digest
SHA256 e64006c0a9046c04cffebef3e8d154fa0c31f574d72c50bb7c3c1d7ac691dcc5
MD5 7d10dbf6f22502aab5a70316e1c02cfb
BLAKE2b-256 07b6cd4275ff9519a0337cf58c4e721f199351aaff6fd6b78ec1a5b7815071ca

See more details on using hashes here.

Provenance

The following attestation bundles were made for nnja_ai-1.0.0.tar.gz:

Publisher: publish.yaml on brightbandtech/nnja-ai

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

File details

Details for the file nnja_ai-1.0.0-py3-none-any.whl.

File metadata

  • Download URL: nnja_ai-1.0.0-py3-none-any.whl
  • Upload date:
  • Size: 17.8 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? Yes
  • Uploaded via: twine/6.1.0 CPython/3.12.9

File hashes

Hashes for nnja_ai-1.0.0-py3-none-any.whl
Algorithm Hash digest
SHA256 71d8858b919f60fa0e3810c6ab8e1431669e23f9c709d2d5907e53c1b2bbb59a
MD5 59f3d27ef722f09f5aeb56eac140e26a
BLAKE2b-256 e0022770f013fe79b3244037ac8757d218c03c25ee1b89a0e2cd217974b7eb9d

See more details on using hashes here.

Provenance

The following attestation bundles were made for nnja_ai-1.0.0-py3-none-any.whl:

Publisher: publish.yaml on brightbandtech/nnja-ai

Attestations: Values shown here reflect the state when the release was signed and may no longer be current.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Depot Continuous Integration Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page