VanderSat API client package
Project description
(Command line) interface to download data batches directly from the VanderSat API
Description
Using this module, one can get data from the VanderSat API using either:
Command line
Python console
Compatible for Linux, Mac and Windows Python >3.6
This package offers an easy interface to the asynchronous endpoints offered by the VanderSat API. However, not all available endpoints can be accessed through this package.
Required packages
click
requests
pandas
click_datetime
joblib
Setting up an environment
If you don’t have an environment yet or would like a new one, use the following line to make a new one using conda
$ conda create -n vds_api -c conda-forge python=3 requests "click>=7" pandas joblib pip
activate it
$ conda activate vds_api
and follow the installation steps
Installation
The package can be installed directly from PyPI. Activate your environment and then install with
$ pip install vds_api_client
With this activated environment one can access the vds cli with
$ vds-api
(If not, your installation did not succeed)
Available CLI commands
$ vds-api
will show all available commands which should include:
grid - download gridded data
info - Show info for this account
test - test connection, credentials and if api is operational
ts - download time-series as csv over points or rois
Calling any of these commands should be done after suppliying credentials:
$ vds api -u [username] -p [password] [command]
And it is always a good idea to start with a test:
$ vds-api -u [username] -p [password] test
Credentials
For each api call using the cli, the credentials need to be supplied. These can be parsed along with the call by typing them explicitly like:
$ vds-api -u [username] -p [password] [command]
However, it is also convenient to store the credentials so they don’t have to be typed each time. Set the environment variables $VDS_USER and $VDS_PASS with the correct values to automatically fill in your credentials.
Impersonation
If a user manages another VanderSat API user account, it can impersonate this user. Through the CLI this can also be done using the --impersonate flag. e.g.
$ vds-api -u manager@company.com -p password --impersonate "user@company.com" [command]
or when credentials were stored already
$ vds-api --impersonate "user@company.com" [command]
Command specifications: info
Get a summary of all user information. The shown information contains the following:
registred user information (name, email, role, etc)
registred products (api-name | product-name)
roi information
All info is shown by default but it is also possible to only show part of it with the following options:
- -u, --user
show user info
- -p, --product_list
show product-list
- -r, --roi
show roi info
E.g. to show all available products, type:
$ vds-api info -p
Command specifications: grid
Get one or multiple gridded data files in GeoTIFF or NetCDF.
See all available options by typing:
$ vds-api grid --help
Required options:
- -p, --product
str // Product api-Name to download, you can specify multiple products by repeating the -p flag
- -lo, --lon_range
float float // Range of longitudes, -lo min max
- -la, --lat_range
float float // Range of latitudes, -la min max
- -dr, --date_range
yyyy-mm-dd yyyy-mm-dd // date range to download separated by a space
Optional options:
- -f, --format
[gtiff|netcdf4] // File format to download, defaults to gtiff
- -n, --n_proc
int // Number of simultaneous calls to the server (default 4, pref <= 8)
- -o, --outfold
str // Path to output the data to (created if it does not exist)
- -v, --verbose
Switch to increase the output messages
- -c, --config_file
str // Path to condiguration file containing pre-defined parameters
- -z, --zipped
Switch to request the data zipped (if n_procs > 1, multiple zip files will be received)
Command specifications: ts
Get one or multiple csv files with time-series.
See all available options by typing:
$ vds-api ts --help
Required options:
- -p, --product
str // Product api-Name to download, you can specify multiple products by repeating the -p flag
- -dr, --date_range
yyyy-mm-dd yyyy-mm-dd // date range to download separated by a space
At least one of the following (yet multiple allowed):
- -ll, --latlon
float float // Latitude-Longitude pair to extract ts, can be multiple by repeating -ll
- -r, --roi
int // Region of interest id that can be referenced at maps.vandersat.com. Repeat -r for multiple
Optional options:
- -f, --format
[csv|json] // File format to download, defaults to csv
- --masked
Switch to also download flagged data
- --av_win
int // Add averaging +/- days window column to output (supply full window)
- --backward
Backward moving average, defaults to –center if –av_win xx is used
- --clim
Switch to include climatology column in output
- -t
int // Rootzone soil moisture parameter (days) (not used with streaming)
- -v, --verbose
Switch to increase the output messages
- -c, --config_file
text // Path to condiguration file containing pre-defined parameters
- -o, --outfold
str // Path to output the data to (created if it does not exist)
V2 CLI Examples
Example usage CLI V2 grid
Get L-band for one month over NL in geotiff with 8 threads
$ vds-api grid -p SM-SMAP-LN-DESC_V003_100 -dr 2015-04-01 2015-04-30 -lo 3 8 -la 50 54 -o SM_L_Data -n 8 -v
Get L+C+X-band for two dates over NL in netcdf
$ vds-api grid -p SM-SMAP-LN-DESC_V003_100 -p SM-AMSR2-C1N-DESC_V003_100 -p SM-AMSR2-XN_V003_100 -f netcdf4 -dr 2016-07-01 2016-07-02 -lo 3.0 8.0 -la 50.0 54.0 -o NCData -v
Example usage CLI V2 ts
Get L-band time-series for a region-of-interest (roi) and a lat-lon pair
$ vds-api ts -p SM-SMAP-LN-DESC_V003_100 -dr 2015-05-01 2020-01-01 -ll 52 4.5 -r 3249 -o tsfold -v
Get time-series with all additional columns
$ vds-api ts -p SM-SMAP-LN-DESC_V003_100 -dr 2015-04-01 2019-01-01 -ll 52 4.5 -o tsfold --masked --av_win 35 --backward --clim -t 20 -v
Example usage Python API
Asynchronous requests can easily be downloaded using the VdsApiV2 class. For downloading of the desired files, the following steps need to be taken:
API v2
- For the version 2 api, three steps have to be taken to download data from the api which are all methods of the VdsApiV2 class:
- Generate a request
Configure gridded data download or time-series download through one of gen_time_series_requests() or gen_gridded_data_request()
- Submit request
After generating all desired URIs, submit these with submit_async_requests() to start the processing of these jobs
- Download files
Get all data using download_async_files()
Initialize class
from vds_api_client import VdsApiV2
# Choose one of the following options to initialize
vds = VdsApiV2('username', 'password')
vds = VdsApiV2() # extract login from $VDS_USER and $VDS_PASS
Impersonate user
When a user manages another account, it can impersonate this managed acount which means that all requests will be done as if the impersonated user has made them
vds = VdsApiV2('manager@company.com', 'password')
# Start impersonation
vds.impersonate('user@company.com')
# do_requests
# End impersonation
vds.forget()
Gridded data example [asynchronous]
Request raster data using the products/<api_name>/gridded-data endpoint
from vds_api_client import VdsApiV2
vds = VdsApiV2()
vds.set_outfold('testdata/tiff') # Created if it does not exist
vds.gen_gridded_data_request(products=['SM-SMAP-LN-DESC_V003_100', 'SM-AMSR2-XN-DESC_V003_100'],
start_date='2015-10-01', end_date='2016-09-30',
lat_min=-3.15, lat_max=-1.5, lon_min=105, lon_max=107,
nrequests=4)
vds.submit_async_requests()
vds.download_async_files()
# Get information on the downloaded files
vds.summary()
Time-series example [asynchronous]
Request time-series data using the products/<api_name>/[point|roi]-time-series endpoints.
from vds_api_client import VdsApiV2
vds = VdsApiV2()
vds.set_outfold('testdata/csv') # Created if it does not exist
vds.gen_time_series_requests(products=['SM-XN_V001_100'],
start_time='2018-01-01', end_time='2018-01-03',
lons=[6.5], lats=[41.5], rois=[527, 811])
vds.submit_async_requests()
vds.download_async_files()
# Get information on the downloaded files
vds.summary()
Notice that the lons and lats are given in a list. When multiple points are defined, the latitude and longitude pairs can be added to the single lists like this:
lons=[6.5, 7.5], lats=[41.5, 45]
and they will be processed in parallel.
Re-download previous requests
Re-download data using previously generated uuids. Note that data is not stored indefinitely, but within 7 days you should be able to re-download your data.
from vds_api_client import VdsApiV2
vds = VdsApiV2()
# Choose from
vds.uuids.append('5742540a-cf87-49dd-a6e7-d484de137324')
vds.queue_uuids_files()
# or
vds.queue_uuids_files(uuids=['57f9950a-4e41-49dd-a6e7-d484de137324'])
Get a single point value
Extract a single value based on a product-coordinate using the products/<api-name>/point-value endpoint
from vds_api_client import VdsApiV2
vds = VdsApiV2()
# Load using the roi-id
val = vds.get_value('SM-XN_V001_100', '2020-04-01', lon=20.6, 40.4)
Load Roi time-series as pandas dataframe [synchronous]
Request roi time-series data using the products/<api_name>/roi-time-series-sync endpoint and load the result as a pandas.DataFrame
from vds_api_client import VdsApiV2
vds = VdsApiV2()
# Load using the roi-id
df1 = vds.get_roi_df('SM-XN_V001_100', 2464, '2016-01-01', '2018-12-31')
# Load using the roi-name
df2 = vds.get_roi_df('SM-XN_V001_100', 'MyArea', '2016-01-01', '2018-12-31')
ROIS
Knowing and using the regions of interest (rois) attached to your account is now easier using the client methods that allow you to filter the rois.
from vds_api_client import VdsApiV2
vds = VdsApiV2()
print(vds.rois)
# ID / DISPLAY # | # Name # | # Area # | # Created at # | # Description # =============================================================================================== 25009 / [X] | Center | 1.063e+09 ha | 2020-08-16 12:49 | Center pixels 25010 / [X] | Right | 9.949e+08 ha | 2020-08-16 12:58 | Right side pixels 25011 / [X] | Bottom | 6.616e+08 ha | 2020-08-16 12:59 | Bottom side pixels 30596 / [ ] | NewName | 9.140e+07 ha | 2020-09-18 07:19 | Same rectangle
Filters
But now, also filters can be applied to select Rois based on a criterium, and give the corresponding ids:
rois_filtered = vds.rois.filter(
min_id=25000, max_id=25020,
area_min=1e8, area_max=1e9,
name_regex='Right|Bottom', description_regex='pixels',
created_before=dt.datetime(2020, 8, 16, 13, 0),
created_after=dt.datetime(2020, 8, 16, 12, 57),
display=True)
print(rois_filtered)
print(rois_filtered.ids_to_list())
# ID / DISPLAY # | # Name # | # Area # | # Created at # | # Description # =============================================================================================== 25010 / [X] | Right | 9.949e+08 ha | 2020-08-16 12:58 | Right side pixels 25011 / [X] | Bottom | 6.616e+08 ha | 2020-08-16 12:59 | Bottom side pixels [9211, 9212]
Geometry
Accessing the geometry is now supported through the geojson property:
roi = vds.rois[25010]
geojson = roi.geojson # Loads geometry from api
print(geojson)
{'type': 'MultiPolygon', 'coordinates': [[[[-5.237732, 66.044796], [-5.237732, 66.956952], [-5.018005, 66.956952], [-5.018005, 66.044796], [-5.237732, 66.044796]]]]}
Updating
Updating an Roi’s metadata is supported through the roi.update method:
roi = vds.rois[30596]
roi.update(name='New name', description='New description', display=False)
print(vds.rois.filter(name_regex='New name'))
# ID / DISPLAY # | # Name # | # Area # | # Created at # | # Description # =============================================================================================== 30596 / [ ] | New name | 9.140e+07 ha | 2020-09-18 07:19 | New description
Deleting
Deleting ROIS from your account is supported through the delete_rois_from_account() method. It expects a list of integers, or a filtered Rois instance. Now we can delete our Rois quite easily like:
vds.delete_rois_from_account(vds.rois.filter(description_regex='Selection to Delete'))
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for vds_api_client-2.1.0-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2fbf8c1b24daff1ec0a24fe6f99e787832c5f67268d4337b901421141cd71545 |
|
MD5 | 5206c85e3d09f8801b611776b614f9da |
|
BLAKE2b-256 | 6c76b1d25930f51c417716419c40d44acbd1e957abfc296fae24d59dda7a1c38 |