ceda-sentinel-dhs-tools 0.1.2

Tools for managing CEDA Sentinel DHS

CEDA Sentinel Relay Hub tools

This codebase provides a set of tools to interact with the CEDA DHS series Data Hub Relay (DHR) based on the ESA/SERCO GSS implementation. These tools will allow interaction with the GSS endpoint on the CEDA DHR and will allow programmatic control of such functions as Producers, Consumers and Ingesters. The DHS GSS system is based upon a set of Docker Containers running within Docker Swarm and has the following components:

• GSS-Ingest is the part of the system which connects to sources (CDSE, DHuS, other GSS, Swift container or directory) to get products and put them into the system (Datastores and Metadatastores)
• GSS-Catalogue is the access point to products stored in the system. Users need to authenticate first and then access to data and metadata.
• GSS-Admin is a REST API used to configure Ingesters, Datastores, DatastoreGroups, MetadataStores, Quotas and their properties.
• GSS-Toolbox is a set of scripts to initialize the database and the index (Solr)
• GSS-Notification notifies a defined end point on product deletion activity

Installation

1. Install the package using pip:

   pip install ceda-sentinel-dhs-tools
   

2. Clone the repo:

   git clone https://github.com/cedadev/ceda-sentinel-dhs-tools.git
   cd ceda_sentinel_dhs_tools
   

3. The templates contain sensitive information and are encrypted using git-crypt. To access them:

   a. Install git-crypt:

   # macOS
   brew install git-crypt
   
   # Ubuntu/Debian
   sudo apt-get install git-crypt
   
   # Windows
   Follow this installation guide (https://www.geeksforgeeks.org/how-to-install-git-crypt-on-windows/)
   

   b. Obtain the encryption key file (git-crypt-key)

   c. Unlock the repository:

   git-crypt unlock /path/to/git-crypt-key
   

4. Configure your local GSS instance by creating a config file (e.g., local_gss_instance.cfg) and add a serviceRootUrl:

   [default]
   serviceRootUrl: http://<your-gss-instance>:8082/gss-admin-api
   

Tools

The tools in this repository will interact with the GSS Admin API and provide users the ability to control the data flow from various sources and where this data is placed to allow CEDA full operational control of the GSS analogous to the previous system (DHuS) synchroniser setup.

These operations are grouped as such:

List existing producer/consumers on local GSS.

---

This script shall list both matched producer and consumer pairs previously created and installed on the target GSS. This script wraps a GET to the respective GSS API endpoints:

• http://:8082/gss-admin-api/producers
• http://:8082/gss-admin-api/consumers/

Command Line

list_producer_consumers -l local_gss_instance.cfg

Where:

-l is the option to define the hub (or otherwise) where the operation shall be submitted. i.e. The local GSS instance we want to see the list of producers on.

-s is an optional string filter to be supplied to match content returned according to original filter.

-t is an optional string filter field defining the number of hours the LCD is behind the current date-time. If the -e option is used this will send output to the email address defined in the -e option. This will allow the user to identify where producer/consumer pairs are falling behind.

-e is an optional string containing an email address to be sent a warning if the -t option is used. Can only be used if the -t option is used.

-f is an option to output to stdout the detailed information associated with the given named producer or consumer i.e. "folder_producer1". The script will render the full JSON returned in a human readable format.

Python

import ceda_sentinel_dhs_tools as cst

cst.list_producer_consumers(
    local_config="path/to/local_gss_instance.cfg",
    filter_string="startswith(Name,'S1')",  # optional
    hours_behind=24,  # optional
    email="[e-mail]",  # optional
    full_details="producer_name"  # optional
)

The output from this should group the matched producers and consumers together with basic information on the associated source URL (source hub/GSS i.e. Colhub2), the GSS instance being used (i.e. srh-services13) , filter parameter and Last Creation Date (LCD) in tabular form.

Create a new producer/consumer pair on the local GSS

---

This script shall allow users to generate a new producer/consumer pair on the target host based on the values within the files associated with the supplied options. Other options shall be used to defined the search string and Last Creation date. Other options may eventually be added to supplant/override other fields within the producer/consumer template.

Command Line

create_producer_consumer -t sentinel_2_producer.template -c colhub.cfg -l local_gss_instance.cfg -F "startswith(Name,'S1')"

Where:

-t is the option to define the template to be used to generate the JSON to POST to the defined instance (in the -l option) i.e. The local GSS instance we want to see the list of producers on.

-c is the option to provide the unencrypted credentials to be used in the POST-ed template to be used in the defined instance (in the -l option).

-l is the option to define the hub (or otherwise) where the operation shall be submitted. i.e. The local GSS instance we want to see the list of producers on.

-F is used to supply a filter string i.e. startswith(Name,'S1')

-L is used to define a LastPublicationDate i.e. 2025-01-15T00:00:00.00. (Optional will use current date-time if not supplied).

Python

import ceda_sentinel_dhs_tools as cst

cst.create_producer_consumer(
    template_base="colhub2_gss",
    config_file="path/to/colhub_example.cfg", 
    local_config="path//local_gss_instance.cfg",
    filter_string="startswith(Name,'S1')",
    last_publication_date="2024-01-15T00:00:00.000Z"  # optional
)

Delete or remove producer/consumers

---

This script shall allow users to remove paired producer/consumers (previously identified in the list tool). It shall use the DELETE operation.

Command Line

delete_producer_consumer -l local_gss_instance.cfg -n <producer_consumer base name>

Python

import ceda_sentinel_dhs_tools as cst

cst.delete_producer_consumer(
    local_config="path/to/local_gss_instance.cfg",
    base_name="my_producer"
)

The script shall parse the HTTP response to report whether this operation was successful

Adjust Quotas

---

Adjust HFS Data stored

---