atmos-validation 2.0.4

A python library containing automated data validation tools for the Atmos Data Service

atmos-validation

A library containing validation checks to be run on hindcast or measurement data to ensure API compliance and standardization.

Atmos is a project to streamline discoverability and data access for weather data sources through APIs. The first step to building a lean and efficient API is converting data to the standard format.

Data Validation

In order to ingest measurement or hindcast data into the Atmos Data Store, each source file needs to pass validation. Validation ensures that the file is compliant with the data conventions specified in Conventions. The standard file formats for source files are NetCDF files for hindcasts and ASCII or NetCDF for measurements.

Measurements by definition contain data for a single geographical location, whereas hindcasts are bigger models containing time series for a multitude of locations in (possibly rotated) grids. A measurement file is therefore expected to contain all of its data in a single file. A hindcast is comprised of a set of NetCDF files - all with the same coordinates, attributes, and variables, - where a single file contains data for a unique time period. Depending on the size of the files, the time separation should be either monthly or yearly. A rule-of-thumb is that if the file size grows more than 4 GB, it should be split into smaller files. See docs or example for how to name hindcast files.

To run validation on NetCDF and ASCII source files, we have built the atmos_validation CLI/library. The documentation below describes these checks, the standard format, and how to run validation using the CLI tool.

Documentation

• Conventions
• ASCII Format
• Running CLI

Examples

• Hindcast netcdfs format example
• Measurement netcdf format example
• Measurement ascii format example

Contributing

We welcome different types of contributions, including code, bug reports, issues, feature requests, and documentation. The preferred method of submitting a contribution is either to make an issue on GitHub or to fork the project on GitHub and make a pull request. In the case of bug reports, please provide a detailed explanation describing how to reproduce before submitting.

Before making your initial commit, please run:

pre-commit install

This sets up a Git hook to automatically run checks (like trufflehog secret scanning) every time you commit. You only need to run this command once — after that, the checks will run automatically.