Machine learning benchmark for lightning prediction with GOES16
Project description
GOES 16 Lightning Count Prediction Benchmark
The GOES 16 Lightning Count Prediction benchmark is a deep learning benchmark for HPC systems used for atmospheric science problems.
Contributors
- David John Gagne
- Bill Anderson
- Gunther Wallach
Requirements
The code is designed to run on Python 3.6 and 3.7. It requires the following Python libraries:
- numpy
- scipy
- pandas
- xarray
- tensorflow>=2.0.0
- scikit-learn
- pyproj
- dask distributed (for data processing)
- ipython
- jupyter (for interactive visualization of neural networks)
Setup from Scratch
-
Install Python 3.7 on your machine. I recommend the Miniconda Python installer available here.
-
Create a benchmark environment:
conda create -n goes16 python=3.7
-
Once the environment is installed activate it on your machine:
source activate goes16
-
Install the needed Python libraries from conda
conda install -c conda-forge --yes \
pip \
ipython \
jupyter \
numpy \
scipy \
matplotlib \
xarray \
netcdf4 \
pandas \
pyyaml \
dask \
distributed \
scikit-learn \
pyproj
-
Make sure the CUDA kernel and CUDA toolkit are installed on your system and know the path and versions.
-
Install the tensorflow-gpu binary (if installing tensorflow 1.15) or tensorflow binary (if tensorflow 2). For more detailed installation instructions visit the tensorflow website.
# If you plan to use tensorflow 2
pip install tensorflow
Run Benchmark Script
- Clone the goes16ci git repository to your home directory.
cd ~
git clone https://github.com/NCAR/goes16ci.git
cd goes16ci
- Install the goes16ci library
pip install .
- Download the GOES16 patch files. You will need about 8 GB free to download and untar the data.
python download_data.py
- Run the benchmark script. The script will output trained neural networks and a yaml file with the timing information for each step.
python goes16_deep_learning_benchmark.py
- If you want to modify the neural network or other properties of the script, you can make a copy of benchmark_config_default.yml and modify it. To run the script with the new config file:
python goes16_deep_learning_benchmark.py -c benchmark_config_default.yml
Setup on Cheyenne/Casper
- Clone the git repo to your home directory
cd ~
git clone https://github.com/NCAR/goes16ci.git
cd goes16ci
- Create a link to the patch data on GLADE
ln -s /glade/p/cisl/aiml/dgagne/goes16_nc/ABI_patches_20190315 data
-
Modify the
goes16_benchmark_casper.sh
script with your account number. -
Submit the benchmark script to the casper queue:
sbatch goes16_benchmark_casper.sh
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for goes16ci-0.1.3b0-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 725dded2c1d4b0cf132bc2a2afdfd44bb640a6956d338231cf08975d14946e15 |
|
MD5 | bd5de4468ffcae52269453dbc8147812 |
|
BLAKE2b-256 | 756f33415c4589d44eaeb7037bc5ceeec9f4383ae3c85db72114404dca97ec56 |