Grid Generator (ggen)

Generates Spectral Element (SE) and Regular Latitude Longitude (RLL) grid meshes and performs conservative remapping between list of meshes. Works with tempestremap and nco under the hood. Logs are appended to log.ggen.

Usage

python ggen.py -h

usage: ggen.py [-h] [-r R] [-f F] [-ind IND] [-out OUT] [-gf GF] [-mf MF]
               [-sd] [-scrip] [-mp] [-ir IR]

optional arguments:
  -h, --help  show this help message and exit
  -r R        Output resolutions (e.g. 16, 30, 64x128, 180x360)
  -f F        File Names (input netcdf file names). Use ' ' when using
              wildcards.
  -ind IND    Input directory (current directory is default).
  -out OUT    Output directory (current directory is default).
  -gf GF      Insert grid file.
  -mf MF      Insert map file.
  -sd         Add a sigleton lev dim.
  -scrip      Generate SCRIP files
  -mp         Multiprocessing
  -ir IR      Input resolutions (e.g. 16, 30, 64x128, 180x360)

General Structure

ggen schematic

Installation

Works with e3sm_unifed environment

On compy:

source /share/apps/E3SM/conda_envs/load_latest_e3sm_unified_compy.sh

On Cori:

source /global/common/software/e3sm/anaconda_envs/load_latest_e3sm_unified_cori-haswell.sh

For others, use the YAML file provided to create a virtual conda enviroment (genv)

conda env create -f environment.yml

And then activate genv to use ggen

conda activate genv

Examples

General use (from command line)

python ggen.py -r <output resolutions> -f <input file names> -ind /input/file/directory -out /output/file/directory

Example log.ggen output

################################## Process Started ##################################

[cmd]: python ggen.py -r 30 -f bc_emission_def.nc -ind /Users/hass877/Work/data_analysis -out /Users/hass877/Work/data_analysis


=== driver init done ===

Specifying input file suppresses resolution.
(Recommended for SE to RLL conversion)

Generating RLL grid metadata

Creating SCRIP file RLL180x360_SCRIP.nc in /Users/hass877/Work/data_analysis

Generated /Users/hass877/Work/data_analysis/RLL180x360_SCRIP.nc

Generated /Users/hass877/Work/data_analysis/RLL180x360_SCRIP.nc

Output Resolution: 30

Generating exodus metadata

Generating pg2 metadata

Creating SCRIP file ne30pg2_SCRIP.nc in /Users/hass877/Work/data_analysis

Generated /Users/hass877/Work/data_analysis/ne30pg2_SCRIP.nc

Generated /Users/hass877/Work/data_analysis/ne30pg2_SCRIP.nc

=== gen_scrips done ===

Input SCRIP:/Users/hass877/Work/data_analysis/RLL180x360_SCRIP.nc
Output SCRIP:/Users/hass877/Work/data_analysis/ne30pg2_SCRIP.nc

[cmd]: ncremap --alg_typ=fv2fv_flx --src_grd=/Users/hass877/Work/data_analysis/RLL180x360_SCRIP.nc --dst_grd=/Users/hass877/Work/data_analysis/ne30pg2_SCRIP.nc --map=/Users/hass877/Work/data_analysis/map_RLL180x360_ne30pg2.nc

Grid(src): /Users/hass877/Work/data_analysis/RLL180x360_SCRIP.nc
Grid(dst): /Users/hass877/Work/data_analysis/ne30pg2_SCRIP.nc


Generated map_RLL180x360_ne30pg2.nc mapping file in /Users/hass877/Work/data_analysis

=== gen_weights done ===

Applying /Users/hass877/Work/data_analysis/map_RLL180x360_ne30pg2.nc on /Users/hass877/Work/data_analysis/bc_emission_def.nc

[cmd]: ncremap --map=/Users/hass877/Work/data_analysis/map_RLL180x360_ne30pg2.nc /Users/hass877/Work/data_analysis/bc_emission_def.nc /Users/hass877/Work/data_analysis/bc_emission_def_RLL180x360_ne30pg2.nc

Input #00: /Users/hass877/Work/data_analysis/bc_emission_def.nc
Map/Wgt  : /Users/hass877/Work/data_analysis/map_RLL180x360_ne30pg2.nc


Generated remapped file /Users/hass877/Work/data_analysis/bc_emission_def_RLL180x360_ne30pg2.nc

=== apply_weights done ===

=== gen_remapped_files done ===

Finished in 4.17 second(s)

################################## Process Finished ##################################
######################################################################################

Using muliprocessing and wildcards

python ggen.py -r 16,32,180x360 -f "*bc*" -ind /compyfs/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne120/ -out /compyfs/hass877/e3sm_scratch/ggen_test -mp
Click to view long log file

log.ggen output

################################## Process Started ##################################

[cmd]: python ggen.py -r 16,32,180x360 -f *bc* -ind /compyfs/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne120/ -out /compyfs/hass877/e3sm_scratch/ggen_test -mp


=== driver init done ===

Specifying input file suppresses resolution.
(Recommended for SE to RLL conversion)

Generating RLL grid metadata

Creating SCRIP file RLL384x576_SCRIP.nc in /compyfs/hass877/e3sm_scratch/ggen_test

Generated /compyfs/hass877/e3sm_scratch/ggen_test/RLL384x576_SCRIP.nc

Generated /compyfs/hass877/e3sm_scratch/ggen_test/RLL384x576_SCRIP.nc

Specifying input file suppresses resolution.
(Recommended for SE to RLL conversion)
/compyfs/hass877/e3sm_scratch/ggen_test/RLL384x576_SCRIP.nc already exists!
 Using it.

Generated /compyfs/hass877/e3sm_scratch/ggen_test/RLL384x576_SCRIP.nc

Output Resolution: 16

Generating exodus metadata

Generating pg2 metadata

Creating SCRIP file ne16pg2_SCRIP.nc in /compyfs/hass877/e3sm_scratch/ggen_test

Generated /compyfs/hass877/e3sm_scratch/ggen_test/ne16pg2_SCRIP.nc

Generated /compyfs/hass877/e3sm_scratch/ggen_test/ne16pg2_SCRIP.nc

Output Resolution: 32

Generating exodus metadata

Generating pg2 metadata

Creating SCRIP file ne32pg2_SCRIP.nc in /compyfs/hass877/e3sm_scratch/ggen_test

Generated /compyfs/hass877/e3sm_scratch/ggen_test/ne32pg2_SCRIP.nc

Generated /compyfs/hass877/e3sm_scratch/ggen_test/ne32pg2_SCRIP.nc

Output Resolution: 180x360

Generating RLL grid metadata

Creating SCRIP file RLL180x360_SCRIP.nc in /compyfs/hass877/e3sm_scratch/ggen_test

Generated /compyfs/hass877/e3sm_scratch/ggen_test/RLL180x360_SCRIP.nc

Generated /compyfs/hass877/e3sm_scratch/ggen_test/RLL180x360_SCRIP.nc

=== gen_scrips done ===

Input SCRIP:/compyfs/hass877/e3sm_scratch/ggen_test/RLL384x576_SCRIP.nc
Output SCRIP:/compyfs/hass877/e3sm_scratch/ggen_test/ne16pg2_SCRIP.nc

[cmd]: ncremap --alg_typ=fv2fv_flx --src_grd=/compyfs/hass877/e3sm_scratch/ggen_test/RLL384x576_SCRIP.nc --dst_grd=/compyfs/hass877/e3sm_scratch/ggen_test/ne16pg2_SCRIP.nc --map=/compyfs/hass877/e3sm_scratch/ggen_test/map_RLL384x576_ne16pg2.nc

Grid(src): /compyfs/hass877/e3sm_scratch/ggen_test/RLL384x576_SCRIP.nc
Grid(dst): /compyfs/hass877/e3sm_scratch/ggen_test/ne16pg2_SCRIP.nc


Generated map_RLL384x576_ne16pg2.nc mapping file in /compyfs/hass877/e3sm_scratch/ggen_test

Input SCRIP:/compyfs/hass877/e3sm_scratch/ggen_test/RLL384x576_SCRIP.nc
Output SCRIP:/compyfs/hass877/e3sm_scratch/ggen_test/ne32pg2_SCRIP.nc

[cmd]: ncremap --alg_typ=fv2fv_flx --src_grd=/compyfs/hass877/e3sm_scratch/ggen_test/RLL384x576_SCRIP.nc --dst_grd=/compyfs/hass877/e3sm_scratch/ggen_test/ne32pg2_SCRIP.nc --map=/compyfs/hass877/e3sm_scratch/ggen_test/map_RLL384x576_ne32pg2.nc

Grid(src): /compyfs/hass877/e3sm_scratch/ggen_test/RLL384x576_SCRIP.nc
Grid(dst): /compyfs/hass877/e3sm_scratch/ggen_test/ne32pg2_SCRIP.nc


Generated map_RLL384x576_ne32pg2.nc mapping file in /compyfs/hass877/e3sm_scratch/ggen_test

Input SCRIP:/compyfs/hass877/e3sm_scratch/ggen_test/RLL384x576_SCRIP.nc
Output SCRIP:/compyfs/hass877/e3sm_scratch/ggen_test/RLL180x360_SCRIP.nc

[cmd]: ncremap --alg_typ=fv2fv_flx --src_grd=/compyfs/hass877/e3sm_scratch/ggen_test/RLL384x576_SCRIP.nc --dst_grd=/compyfs/hass877/e3sm_scratch/ggen_test/RLL180x360_SCRIP.nc --map=/compyfs/hass877/e3sm_scratch/ggen_test/map_RLL384x576_RLL180x360.nc

Grid(src): /compyfs/hass877/e3sm_scratch/ggen_test/RLL384x576_SCRIP.nc
Grid(dst): /compyfs/hass877/e3sm_scratch/ggen_test/RLL180x360_SCRIP.nc


Generated map_RLL384x576_RLL180x360.nc mapping file in /compyfs/hass877/e3sm_scratch/ggen_test

=== gen_weights done ===

Applied multiprocessing.

Applied multiprocessing.

Applying /compyfs/hass877/e3sm_scratch/ggen_test/map_RLL384x576_ne16pg2.nc on /compyfs/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne120/cmip6_mam4_bc_a4_elev_1850-2014_c20191108.nc

[cmd]: ncremap --map=/compyfs/hass877/e3sm_scratch/ggen_test/map_RLL384x576_ne16pg2.nc /compyfs/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne120/cmip6_mam4_bc_a4_elev_1850-2014_c20191108.nc /compyfs/hass877/e3sm_scratch/ggen_test/cmip6_mam4_bc_a4_elev_1850-2014_c20191108_RLL384x576_ne16pg2.nc


Applied multiprocessing.

Applying /compyfs/hass877/e3sm_scratch/ggen_test/map_RLL384x576_ne16pg2.nc on /compyfs/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne120/cmip6_mam4_bc_a4_surf_1850-2014_c20191108.nc

[cmd]: ncremap --map=/compyfs/hass877/e3sm_scratch/ggen_test/map_RLL384x576_ne16pg2.nc /compyfs/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne120/cmip6_mam4_bc_a4_surf_1850-2014_c20191108.nc /compyfs/hass877/e3sm_scratch/ggen_test/cmip6_mam4_bc_a4_surf_1850-2014_c20191108_RLL384x576_ne16pg2.nc


Applied multiprocessing.

Applying /compyfs/hass877/e3sm_scratch/ggen_test/map_RLL384x576_ne32pg2.nc on /compyfs/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne120/cmip6_mam4_bc_a4_elev_1850-2014_c20191108.nc

[cmd]: ncremap --map=/compyfs/hass877/e3sm_scratch/ggen_test/map_RLL384x576_ne32pg2.nc /compyfs/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne120/cmip6_mam4_bc_a4_elev_1850-2014_c20191108.nc /compyfs/hass877/e3sm_scratch/ggen_test/cmip6_mam4_bc_a4_elev_1850-2014_c20191108_RLL384x576_ne32pg2.nc


Applied multiprocessing.

Applying /compyfs/hass877/e3sm_scratch/ggen_test/map_RLL384x576_ne32pg2.nc on /compyfs/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne120/cmip6_mam4_bc_a4_surf_1850-2014_c20191108.nc

[cmd]: ncremap --map=/compyfs/hass877/e3sm_scratch/ggen_test/map_RLL384x576_ne32pg2.nc /compyfs/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne120/cmip6_mam4_bc_a4_surf_1850-2014_c20191108.nc /compyfs/hass877/e3sm_scratch/ggen_test/cmip6_mam4_bc_a4_surf_1850-2014_c20191108_RLL384x576_ne32pg2.nc


Applied multiprocessing.

Applying /compyfs/hass877/e3sm_scratch/ggen_test/map_RLL384x576_RLL180x360.nc on /compyfs/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne120/cmip6_mam4_bc_a4_elev_1850-2014_c20191108.nc

[cmd]: ncremap --map=/compyfs/hass877/e3sm_scratch/ggen_test/map_RLL384x576_RLL180x360.nc /compyfs/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne120/cmip6_mam4_bc_a4_elev_1850-2014_c20191108.nc /compyfs/hass877/e3sm_scratch/ggen_test/cmip6_mam4_bc_a4_elev_1850-2014_c20191108_RLL384x576_RLL180x360.nc


Applying /compyfs/hass877/e3sm_scratch/ggen_test/map_RLL384x576_RLL180x360.nc on /compyfs/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne120/cmip6_mam4_bc_a4_surf_1850-2014_c20191108.nc

[cmd]: ncremap --map=/compyfs/hass877/e3sm_scratch/ggen_test/map_RLL384x576_RLL180x360.nc /compyfs/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne120/cmip6_mam4_bc_a4_surf_1850-2014_c20191108.nc /compyfs/hass877/e3sm_scratch/ggen_test/cmip6_mam4_bc_a4_surf_1850-2014_c20191108_RLL384x576_RLL180x360.nc

Input #00: /compyfs/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne120/cmip6_mam4_bc_a4_surf_1850-2014_c20191108.nc
Map/Wgt  : /compyfs/hass877/e3sm_scratch/ggen_test/map_RLL384x576_ne16pg2.nc


Generated remapped file /compyfs/hass877/e3sm_scratch/ggen_test/cmip6_mam4_bc_a4_surf_1850-2014_c20191108_RLL384x576_ne16pg2.nc

=== apply_weights done ===
Input #00: /compyfs/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne120/cmip6_mam4_bc_a4_surf_1850-2014_c20191108.nc
Map/Wgt  : /compyfs/hass877/e3sm_scratch/ggen_test/map_RLL384x576_ne32pg2.nc


Generated remapped file /compyfs/hass877/e3sm_scratch/ggen_test/cmip6_mam4_bc_a4_surf_1850-2014_c20191108_RLL384x576_ne32pg2.nc

=== apply_weights done ===
Input #00: /compyfs/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne120/cmip6_mam4_bc_a4_surf_1850-2014_c20191108.nc
Map/Wgt  : /compyfs/hass877/e3sm_scratch/ggen_test/map_RLL384x576_RLL180x360.nc


Generated remapped file /compyfs/hass877/e3sm_scratch/ggen_test/cmip6_mam4_bc_a4_surf_1850-2014_c20191108_RLL384x576_RLL180x360.nc

=== apply_weights done ===
Input #00: /compyfs/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne120/cmip6_mam4_bc_a4_elev_1850-2014_c20191108.nc
Map/Wgt  : /compyfs/hass877/e3sm_scratch/ggen_test/map_RLL384x576_ne16pg2.nc


Generated remapped file /compyfs/hass877/e3sm_scratch/ggen_test/cmip6_mam4_bc_a4_elev_1850-2014_c20191108_RLL384x576_ne16pg2.nc

=== apply_weights done ===
Input #00: /compyfs/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne120/cmip6_mam4_bc_a4_elev_1850-2014_c20191108.nc
Map/Wgt  : /compyfs/hass877/e3sm_scratch/ggen_test/map_RLL384x576_ne32pg2.nc


Generated remapped file /compyfs/hass877/e3sm_scratch/ggen_test/cmip6_mam4_bc_a4_elev_1850-2014_c20191108_RLL384x576_ne32pg2.nc

=== apply_weights done ===
Input #00: /compyfs/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne120/cmip6_mam4_bc_a4_elev_1850-2014_c20191108.nc
Map/Wgt  : /compyfs/hass877/e3sm_scratch/ggen_test/map_RLL384x576_RLL180x360.nc


Generated remapped file /compyfs/hass877/e3sm_scratch/ggen_test/cmip6_mam4_bc_a4_elev_1850-2014_c20191108_RLL384x576_RLL180x360.nc

=== apply_weights done ===

=== gen_remapped_files done ===

Finished in 348.79 second(s)

################################## Process Finished ##################################
######################################################################################

Submit as a batch job

Example batch script

#!/bin/bash -l
#SBATCH --job-name=ggen
#SBATCH --output=ggen.o%j
#SBATCH --account=project
#SBATCH --nodes=1
#SBATCH --time=00:30:00
#SBATCH --partition=short

source /share/apps/E3SM/conda_envs/load_latest_e3sm_unified_compy.sh
python ggen.py -r 16,32,180x360 -f "*bc*" -ind /compyfs/inputdata/atm/cam/chem/trop_mozart_aero/emis/DECK_ne120/ -out /compyfs/hass877/e3sm_scratch/ggen_test -mp

References

[1] Ullrich, Paul A and Taylor, Mark A (2015). Arbitrary-order conservative and consistent remapping and a theory of linear maps: Part I. Monthly Weather Review, 143, 2419-2440.

[2] Ullrich, Paul A and Devendran, Dharshi and Johansen, Hans (2016). Arbitrary-order conservative and consistent remapping and a theory of linear maps: Part II. Monthly Weather Review, 144, 1529-1549.

[3] Zender, Charles S (2008). Analysis of self-describing gridded geoscience data with netCDF Operators (NCO). Environmental Modelling & Software, 23, 1338-1342

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for thassan from gravatar.com thassan

Credits

Author: Taufiq Hassan

License

MIT License (MIT)

Classifiers

Metadata

Project links

Key dates

PyPI data
Data sourced directly from PyPI's database.
  • Released:
Latest release

1 maintainer

PyPI data
Data sourced directly from PyPI's database.
Avatar for thassan from gravatar.com thassan

Credits

Author: Taufiq Hassan

License

MIT License (MIT)

Classifiers

Release files for ggen 1.1.0

For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.

Source distribution (sdist)

Source distribution for ggen 1.1.0
File Size Uploaded Provenance Actions
ggen-1.1.0.tar.gz 19.6 kB Without provenance Details for ggen-1.1.0.tar.gz

Built distribution (wheel)

Table of built distributions (wheels) for ggen 1.1.0
File Interpreter ABI Platform Actions
ggen-1.1.0-py3-none-any.whl 21.1 kB Without provenance Python 3 none any Details for ggen-1.1.0-py3-none-any.whl

Total release size: 40.7 kB

Release files / ggen-1.1.0.tar.gz

Download URL ggen-1.1.0.tar.gz
Size 19.6 kB
Tags Source
SHA-256 checksum
How to use checksums 		0831f95ecfa3a8fff18d4f9a453b36c23828fcf2fc76160971e50cc3162e5475
BLAKE2b-256 checksum
How to use checksums 		3cfaf030cef10932629b9540841a0b7cf0a455940a90071691c8656a70e10954
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing? 		No
Uploaded via twine/4.0.2 CPython/3.10.11

Release files / ggen-1.1.0-py3-none-any.whl

Download URL ggen-1.1.0-py3-none-any.whl
Size 21.1 kB
Tags Python 3
SHA-256 checksum
How to use checksums 		0bad46a40b06e8264b80302b780ac1559b5a3589d93874140eb1a9746b9271a7
BLAKE2b-256 checksum
How to use checksums 		5db06387844c6ce59bf94b08777023885151aaa15e00857d991dfbe933228916
Upload date
Uploaded using Trusted Publishing?
What is trusted publishing? 		No
Uploaded via twine/4.0.2 CPython/3.10.11

Security for ggen 1.1.0

Provenance

Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.

No provenance found

None of the files in this release were uploaded with provenance metadata.

A note on package security

PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.

To learn more about protecting yourself from malicious actors, visit our package security documentation.

Release history Release notifications | RSS feed

This release

1.1.0 This release

2 release files No provenance

1.0.0

2 release files No provenance

Anthropic, PBC Visionary sponsor Bloomberg Visionary sponsor Hudson River Trading Visionary sponsor Meta Visionary sponsor NVIDIA Visionary sponsor Microsoft Sustainability sponsor Depot Continuous Integration AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Sentry Error logging StatusPage Status page