An ML practitioners utility for working with SDO data.
Project description
sdo-cli
A practitioner's utility for working with SDO data.
Setup
Setup Virtual Environment and install sdo-cli
.
make setup
make install
Usage
A small helper toolkit for downloading and working with SDO data complementing sunpy by giving illustrative examples how to solve tasks. The data is loaded from the Image Parameter dataset which is the result of [1].
TLDR;
How to use sdo-cli
:
Usage: sdo-cli data [OPTIONS] COMMAND [ARGS]...
Options:
--help Show this message and exit.
Commands:
download Loads a set of SDO images between start and end from the Georgia
State University Data Lab API
patch Generates patches from a set of images
resize Generates a set of resized images
Examples:
Download images:
sdo-cli data download --path='./data/aia_171_2012' --start='2012-03-07T00:02:00' --end='2012-03-07T00:40:00' --freq='6min' --wavelength='171'
Resize images:
sdo-cli data resize --path='./data/aia_171_2012' --targetpath='./data/aia_171_2012_256' --wavelength='171' --size=256
Patch images:
sdo-cli data patch --path='./data/aia_171_2012_256' --targetpath='./data/aia_171_2012_256_patches' --wavelength='171' --size=32
Loading Events from HEK:
docker-compose up
sdo-cli events get --start="2012-01-01T00:00:00" --end="2012-01-02T23:59:59" --event-type="AR"
SOoD Anomaly Detection
Under src/sood
a Solar Out-of-Distribution model based on a context-encoding variational autoencoder by Zimmerer et al. [2] is implemented. The model makes use of the model-internal latent representation deviations to end up with a more expressive reconstruction error and allows anomaly detection on both a sample as well as a pixel level.
A full Anomaly Detection pipeline can be examined in the example notebook notebooks/e2ePipeline.ipynb
. For this start jupyter:
make notebook
Troubleshooting
Tensorflow only works with Python versions < 3.9.
brew install pyenv
echo 'eval "$(pyenv init -)"' >> ~/.bash_profile
source ~/.bash_profile
pyenv install 3.8.0
Also refer to this link.
References
- [1] Ahmadzadeh, Azim, Dustin J. Kempton, and Rafal A. Angryk. "A Curated Image Parameter Data Set from the Solar Dynamics Observatory Mission." The Astrophysical Journal Supplement Series 243.1 (2019): 18.
- [2] Zimmerer, David, et al. "Context-encoding variational autoencoder for unsupervised anomaly detection." arXiv preprint arXiv:1812.05941 (2018).
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.