Satip provides the functionality necessary for retrieving, transforming and storing EUMETSAT data
Project description
Satip
Satip is a library for satellite image processing, and provides all of the functionality necessary for retrieving, transforming and storing EUMETSAT data
Installation
To install the satip
library please run:
pip install satip
Or if you're working in the development environment you can run the following from the directory root:
pip install -e .
Other dependencies
pbzip2
is used to compress files and should be installed.
Windows: http://gnuwin32.sourceforge.net/packages/bzip2.htm
Mac: brew install bzip2
Linux: sudo apt-get install -y pbzip2
Notebooks
Id | Directory | Number | Description | Maintainer |
---|---|---|---|---|
Utilities | nbs | 00 | Code for keeping the repository tidy | Ayrton Bourn |
EUMETSAT | nbs | 01 | Development of the API wrapper for ems | Ayrton Bourn |
Reprojection | nbs | 02 | Development of the reprojection operator | Ayrton Bourn |
Zarr | nbs | 03 | Development of wrappers for loading/saving to Zarr | Ayrton Bourn |
GCP | nbs | 04 | Development of GCP interface wrappers | Laurence Watson |
Pipeline | nbs | 05 | Development of the pipeline processes | Ayrton Bourn |
CI/CD | nbs | 06 | Development of CI/CD helper functions | Ayrton Bourn |
Downloading | nbs | 101 | Guidance for using the ems download manager | Ayrton Bourn |
Reprojecting | nbs | 102 | Guidance for using the reprojection operator | Ayrton Bourn |
Loading | nbs | 103 | Guidance for retrieving saved data from Zarr | Ayrton Bourn |
Documentation | docs | - | Automated generation of docs from notebooks | Ayrton Bourn |
Development Set-Up
To create a new environment you can follow the following code blocks or run the setup_env
batch script located in the batch_scripts directory.
git clone
conda env create -f environment.yml
conda activate sat_image_processing
We'll also install Jupyter lab interactive plotting for matplotlib
See the jupyter-matplotlib docs for more info. The short version is to run these commands from within the sat_image_processing
env:
jupyter labextension install @jupyter-widgets/jupyterlab-manager
jupyter labextension install jupyter-matplotlib
Google Cloud Platform
You will need to be authenticated with Google Cloud Platform (GCP) to use much of the library - currently it is fairly tightly coupled to storage and BigQuery for saving metadata.
Install the GCP SDK and run:
gcloud auth login
and to "generate credentials for client libraries"
https://googleapis.dev/python/google-api-core/latest/auth.html
gcloud auth application-default login
Publishing to PyPi
To automatically publish the satip
module to PyPi you can add a GitHub tag of the form v.X.X.X
, where the version number must be higher than the current one. This will also generate a GitHub release at the same time.
To manually publish the satip
module to PyPi simply run the following from the batch_scripts directory
pypi_publish <anaconda_dir>
Where <anaconda_dir>
is the path to your anaconda directory - e.g. C:\Users\User\anaconda3
When prompted you should enter your PyPi username and password
After this you will be able to install the latest version of satip using pip install satip
Pipeline
To run the dagster pipeline you can use: dagster pipeline execute -m satip.mario -c pipeline_inputs.yaml
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.