Skip to main content

VanderSat API client package

Project description

https://badge.fury.io/py/vds-api-client.svg https://travis-ci.org/vandersat/vds-api-client.svg?branch=master https://coveralls.io/repos/github/vandersat/vds-api-client/badge.svg?branch=master

(Command line) interface to download data batches directly from the VanderSat API

Description

Using this module, one can get data from the VanderSat API using either:

Compatible for Linux, Mac and Windows

Python >= 3.6

This package offers an easy interface to the asynchronous endpoints offered by the VanderSat API. However, not all available endpoints can be accessed through this package.

Installation

Required packages

  • click
  • requests
  • pandas
  • click_datetime
  • joblib

Setting up an environment

If you don’t have an environment yet or would like a new one, use the following line to make a new one using conda

$ conda create -n vds_api -c conda-forge python=3 requests "click>=7" pandas joblib pip

activate it

$ conda activate vds_api

and follow the installation steps

Installing the client

The package can be installed directly from PyPI. Activate your environment and then install with

$ pip install vds_api_client

With this activated environment one can access the vds cli with

$ vds-api

(If not, your installation did not succeed)

Command line interface

Available CLI commands

$ vds-api

will show all available commands which should include:

  • grid - download gridded data
  • info - Show info for this account
  • test - test connection, credentials and if api is operational
  • ts - download time-series as csv over points or rois

Calling any of these commands should be done after suppliying credentials:

$ vds api -u [username] -p [password] [command]

And it is always a good idea to start with a test:

$ vds-api -u [username] -p [password] test

Credentials

For each api call using the cli, the credentials need to be supplied. These can be parsed along with the call by typing them explicitly like:

$ vds-api -u [username] -p [password] [command]

However, it is also convenient to store the credentials so they don’t have to be typed each time. Set the environment variables $VDS_USER and $VDS_PASS with the correct values to automatically fill in your credentials.

Impersonation

If a user manages another VanderSat API user account, it can impersonate this user. Through the CLI this can also be done using the --impersonate flag. e.g.

$ vds-api -u manager@company.com -p password --impersonate "user@company.com" [command]

or when credentials were stored already

$ vds-api --impersonate "user@company.com" [command]

Note

From this point on, the credentials don’t have to be parsed explicitly anymore thus the syntax reduces to:

$ vds-api [command]

Command specific options

Use the help function to retrieve all options for the command line interface.

$ vds-api [command] --help

Example usage CLI V2 grid

Get L-band for one month over NL in geotiff with 8 threads

$ vds-api grid -p SM-SMAP-LN-DESC_V003_100 -dr 2015-04-01 2015-04-30 -lo 3 8 -la 50 54 -o SM_L_Data -n 8 -v

Get L+C+X-band for two dates over NL in netcdf

$ vds-api grid -p SM-SMAP-LN-DESC_V003_100 -p SM-AMSR2-C1N-DESC_V003_100 -p SM-AMSR2-XN_V003_100 -f netcdf4 -dr 2016-07-01 2016-07-02 -lo 3.0 8.0 -la 50.0 54.0 -o NCData -v

Example usage CLI V2 ts

Get L-band time-series for a region-of-interest (roi) and a lat-lon pair

$ vds-api ts -p SM-SMAP-LN-DESC_V003_100 -dr 2015-05-01 2020-01-01 -ll 52 4.5 -r 3249 -o tsfold -v

Get time-series with all additional columns

$ vds-api ts -p SM-SMAP-LN-DESC_V003_100 -dr 2015-04-01 2019-01-01 -ll 52 4.5 -o tsfold --masked --av_win 35 --backward --clim -t 20 -cov -v

Example usage Python API

Asynchronous requests can easily be downloaded using the VdsApiV2 class. For downloading of the desired files, the following steps need to be taken:

API v2

For the version 2 api, three steps have to be taken to download data from the api which are all methods of the VdsApiV2 class:
  1. Generate a request
    Configure gridded data download or time-series download through one of gen_time_series_requests() or gen_gridded_data_request()
  2. Submit request
    After generating all desired URIs, submit these with submit_async_requests() to start the processing of these jobs
  3. Download files
    Get all data using download_async_files()

Initialize class

from vds_api_client import VdsApiV2

# Choose one of the following options to initialize
vds = VdsApiV2('username', 'password')
vds = VdsApiV2()  # extract login from $VDS_USER and $VDS_PASS

Impersonate user

When a user manages another account, it can impersonate this managed acount which means that all requests will be done as if the impersonated user has made them

vds = VdsApiV2('manager@company.com', 'password')

# Start impersonation
vds.impersonate('user@company.com')

# do_requests

# End impersonation
vds.forget()

Gridded data example [asynchronous]

Request raster data using the products/<api_name>/gridded-data endpoint

from vds_api_client import VdsApiV2

vds = VdsApiV2()

vds.set_outfold('testdata/tiff')  # Created if it does not exist
vds.gen_gridded_data_request(products=['SM-SMAP-LN-DESC_V003_100', 'SM-AMSR2-XN-DESC_V003_100'],
                             start_date='2015-10-01', end_date='2016-09-30',
                             lat_min=-3.15, lat_max=-1.5, lon_min=105, lon_max=107,
                             nrequests=4)
vds.submit_async_requests()
vds.download_async_files()

# Get information on the downloaded files
vds.summary()

Time-series example [asynchronous]

Request time-series data using the products/<api_name>/[point|roi]-time-series endpoints.

from vds_api_client import VdsApiV2
vds = VdsApiV2()

vds.set_outfold('testdata/csv')  # Created if it does not exist
vds.gen_time_series_requests(products=['SM-XN_V001_100'],
                             start_time='2018-01-01', end_time='2018-01-03',
                             lons=[6.5], lats=[41.5], rois=[527, 811])
vds.submit_async_requests()
vds.download_async_files()

# Get information on the downloaded files
vds.summary()

Notice that the lons and lats are given in a list. When multiple points are defined, the latitude and longitude pairs can be added to the single lists like this:

lons=[6.5, 7.5], lats=[41.5, 45]

and they will be processed in parallel.

Re-download previous requests

Re-download data using previously generated uuids. Note that data is not stored indefinitely, but within 7 days you should be able to re-download your data.

from vds_api_client import VdsApiV2
vds = VdsApiV2()

# Choose from
vds.uuids.append('5742540a-cf87-49dd-a6e7-d484de137324')
vds.queue_uuids_files()
# or
vds.queue_uuids_files(uuids=['57f9950a-4e41-49dd-a6e7-d484de137324'])

Get a single point value

Extract a single value based on a product-coordinate using the products/<api-name>/point-value endpoint

from vds_api_client import VdsApiV2

vds = VdsApiV2()

# Load using the roi-id
val = vds.get_value('SM-XN_V001_100', '2020-04-01', lon=20.6, 40.4)

Load Roi time-series as pandas dataframe [synchronous]

Request roi time-series data using the products/<api_name>/roi-time-series-sync endpoint and load the result as a pandas.DataFrame

from vds_api_client import VdsApiV2

vds = VdsApiV2()

# Load using the roi-id
df1 = vds.get_roi_df('SM-XN_V001_100', 2464, '2016-01-01', '2018-12-31')

# Load using the roi-name
df2 = vds.get_roi_df('SM-XN_V001_100', 'MyArea', '2016-01-01', '2018-12-31')

ROIS

Knowing and using the regions of interest (rois) attached to your account is now easier using the client methods that allow you to filter the rois.

from vds_api_client import VdsApiV2

vds = VdsApiV2()

print(vds.rois)
 # ID / DISPLAY # |  # Name #  |   # Area #   |  # Created at #  |       # Description #
===============================================================================================
   25009  /  [X]  | Center     | 1.063e+09 ha | 2020-08-16 12:49 | Center pixels
   25010  /  [X]  | Right      | 9.949e+08 ha | 2020-08-16 12:58 | Right side pixels
   25011  /  [X]  | Bottom     | 6.616e+08 ha | 2020-08-16 12:59 | Bottom side pixels
   30596  /  [ ]  | NewName    | 9.140e+07 ha | 2020-09-18 07:19 | Same rectangle

Filters

But now, also filters can be applied to select Rois based on a criterium, and give the corresponding ids:

rois_filtered = vds.rois.filter(
    min_id=25000, max_id=25020,
    area_min=1e8, area_max=1e9,
    name_regex='Right|Bottom', description_regex='pixels',
    created_before=dt.datetime(2020, 8, 16, 13, 0),
    created_after=dt.datetime(2020, 8, 16, 12, 57),
    display=True)
print(rois_filtered)
print(rois_filtered.ids_to_list())
 # ID / DISPLAY # |  # Name #  |   # Area #   |  # Created at #  |       # Description #
===============================================================================================
   25010  /  [X]  | Right      | 9.949e+08 ha | 2020-08-16 12:58 | Right side pixels
   25011  /  [X]  | Bottom     | 6.616e+08 ha | 2020-08-16 12:59 | Bottom side pixels

[25010, 25011]

Geometry

Accessing the geometry is now supported through the geojson property:

roi = vds.rois[25010]
geojson = roi.geojson  # Loads geometry from api
print(geojson)

{'type': 'MultiPolygon', 'coordinates': [[[[-5.237732, 66.044796], [-5.237732, 66.956952], [-5.018005, 66.956952], [-5.018005, 66.044796], [-5.237732, 66.044796]]]]}

Updating

Updating an Roi’s metadata is supported through the roi.update method:

roi = vds.rois[30596]
roi.update(name='New name', description='New description', display=False)
print(vds.rois.filter(name_regex='New name'))
 # ID / DISPLAY # |  # Name #  |   # Area #   |  # Created at #  |       # Description #
===============================================================================================
   30596  /  [ ]  | New name   | 9.140e+07 ha | 2020-09-18 07:19 | New description

Deleting

Deleting ROIS from your account is supported through the delete_rois_from_account() method. It expects a list of integers, or a filtered Rois instance. Now we can delete our Rois quite easily like:

vds.delete_rois_from_account(vds.rois.filter(description_regex='Selection to Delete'))

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for vds-api-client, version 2.1.3
Filename, size File type Python version Upload date Hashes
Filename, size vds_api_client-2.1.3-py2.py3-none-any.whl (26.3 kB) File type Wheel Python version py2.py3 Upload date Hashes View
Filename, size vds_api_client-2.1.3.tar.gz (41.5 kB) File type Source Python version None Upload date Hashes View

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring DigiCert DigiCert EV certificate Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page