Satip
⚠️ Note this repo is deprecated in favour of satellite-consumer
Satip is a library for satellite image processing, and provides all of the functionality necessary for retrieving, and storing EUMETSAT data
Installation
To install the satip library please run:
pip install satip
Or if you're working in the development environment you can run the following from the directory root:
pip install -e .
Conda
Or, if you want to use conda from a cloned Satip repository:
conda env create -f environment.yml
conda activate satip
pip install -e .
If you plan to work on the development of Satip then also consider installing these development tools:
conda install pytest flake8 jedi mypy black pre-commit
pre-commit install
Development Environment
In order to contribute:
- it's recommended that you use a Linux-based OS. This is currently used for all CI/CD testing, production, and development.
- At the time of writing (21-Dec-23), the Python version used is 3.11 with work being done to update to Python 3.12. This is subject to updates over time.
Operation
Getting your own API key
In order to contribute to development or just test-run some scripts, you will need your own Eumetsat-API-key. Please follow these steps:
- Go to https://eoportal.eumetsat.int and register an account.
- You can log in and go to https://data.eumetsat.int/ to check available data services. From there go to your profile and choose the option "API key" or go to https://api.eumetsat.int/api-key/ directly.
- Please make sure that you added the key and secret to your user's environment variables.
Downloading EUMETSAT Data
We have moved this to here
Converting Native files to Zarr
scripts/convert_native_to_zarr.py converts EUMETSAT .nat files to Zarr datasets, using very mild lossy JPEG-XL compression. (JPEG-XL is the "new kid on the block" of image compression algorithms). JPEG-XL makes the files about a quarter the size of the equivalent bz2 compressed files, whilst the images are visually indistinguishable. JPEG-XL cannot represent NaNs so NaNs. JPEG-XL understands float32 values in the range [0, 1]. NaNs are encoded as the value 0.025. All "real" values are in the range [0.075, 1]. We leave a gap between "NaNs" and "real values" because there is very slight "ringing" around areas of constant value (see this comment for more details). Use satip.jpeg_xl_float_with_nans.JpegXlFloatWithNaNs to decode the satellite data. This class will reconstruct the NaNs and rescale the data to the range [0, 1].
Running in Production
The live service uses app.py as the entrypoint for running the live data download for OCF's forecasting service, and has a few configuration options, configurable by command line argument or environment variable.
--api-key or API_KEY is the EUMETSAT API key
--api-secret or API_SECRET is the EUMETSAT API secret
--save-dir or SAVE_DIR is the top level directory to save the output files, a latest subfolder will be added to that directory to contain the latest data
--history or HISTORY is the amount of history timesteps to use in the latest.zarr files
--db-url or DB_URL is the URL to the database to save to when a run has finished
--use-rescaler or USE_RESCALER tells whether to rescale the satellite data to between 0 and 1 or not when saving to disk. Primarily used as backwards compatibility for the current production models, all new training and production Zarrs should use the rescaled data.
--use-iodc or USE_IODC is an option to get the IODC satellite data
Testing
To run tests, simply run pytest . from the root of the repository. To generate the test plots, run python scripts/generate_test_plots.py.
Environmental Variables
Some tests require environmental variables to be set that would be passed in by command line argument when running the code in production. These are as follows:
EUMETSAT_USER_KEY: the EUMETSAT API keyEUMETSAT_USER_SECRET: the EUMETSAT API secret
These can be added using the export command in your shell environment. To add these permanently, the export statements can be added to the configuration file for the shell environment (e.g. "~/.bashrc" if using bash).
Contributors ✨
Thanks goes to these wonderful people (emoji key):
 Jacob Bieker 💻 |
 Jack Kelly 💻 |
 Ayrton Bourn 💻 |
 Laurence Watson 💻 |
 Notger Heinz 📖 |
 Peter Dudfield 📖 |
 Azah Norbline 💻 |
 Tom Pughe 💻 |
 Zhenbang Feng 💻 |
 jsbaasi 💻 |
 Suleman Karigar 💻 |
 Richa 💻 |
 Nathan Simpson 🐛 |
 peach280 🚧 |
This project follows the all-contributors specification. Contributions of any kind welcome!
3 maintainers
Metadata
3 maintainers
Release files for satip 2.12.41
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Source distribution (sdist)
| File | Size | Uploaded | |
|---|---|---|---|
| satip-2.12.41.tar.gz | 35.6 kB | Details |
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Reset |
|---|---|---|---|---|
| satip-2.12.41-py3-none-any.whl | Python 3 | none | any | Details |
Total release size: 68.2 kB
Release files / satip-2.12.41.tar.gz
| Download URL | satip-2.12.41.tar.gz |
|---|---|
| Size | 35.6 kB |
| Tags | Source |
|
SHA-256 checksum How to use checksums |
4bb9baf7764f961627ffce218c2ded5732e6838224bb438cec4a2f1c594eecbb
|
|
BLAKE2b-256 checksum How to use checksums |
72632e661c4fca17f4bb9b96584a1fa4f7ee8cb2c0f290bed0d2145df77ef309
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.9.23
|
Release files / satip-2.12.41-py3-none-any.whl
| Download URL | satip-2.12.41-py3-none-any.whl |
|---|---|
| Size | 32.6 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
1183132a5afdb0d2aa23983593a6a62590723a37f64ed19d6139a93e83bfb722
|
|
BLAKE2b-256 checksum How to use checksums |
d11bd3772e7dc9ebdb779bdc197bacdf6c5fd64c841e57518daf63d52e4d7657
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/6.1.0 CPython/3.9.23
|
