asf_metadata

This repo contains all the Python 3.8 scripts related to manage ASF metadata. These are currently mostly dealing with ISO metadata.

Table of Contents

• ISO template
• Excel spreadsheet
• ISO metdata functions
• ISO metadata tools

ISO template

ISO metadata templates can be best maintained in a more user-friendly environment. I have settled for an Excel spreadsheet to achieve that.

The template mechanism has three different flows. All use an Excel spreadsheet to define the structure of the ISO metadata.

ISO metadata template generation

The ISO metadata's structure is stored in an Excel spreadsheet and contains identifiers for each metadata parameter that is variable. The structure and variables form the template that we can store in two formats: XML and JSON.

Format conversion

In order to convert the metadata between various formats, we need to know the structure of it. That is saved within an Excel spreadsheet.

This functionality is something I anticipate using for the NISAR mission, for which we only get metadata in ISO XML format. Using the structure from the spreadsheet allows to painlessly deal with the namespaces within the XML files.

ISO metadata generation

For the generation of ISO metadata, we need the structure from the Excel spreadsheet. The template parameters need to be exchanged by their actual values. This will be achieved by extracting the necessary information based on the product files, replacing the functionality that existed in the old MapReady framework.

Excel spreadsheet

The Excel spreadsheet has a number of worksheets.

ISO metadata structure

This worksheet provides the relevant parts of the ISO metadata, most importantly its structure and the namespaces used within that structure.

Namespaces

One of the biggest nuisances about ISO XML metadata is the namespaces.

This worksheet links each parameter to its respective namespace.

This worksheet defines the URLs for the individual namespaces needed in the XML declarations.

ISO metadata functions

I developed a number of functions that perform the various tasks outline above.

Extracting information from templates

The template information in the Excel spreadsheet is extracted (using the isoTemplate2lists function) into a number of lists: the flattened template structure, the individual metadata parameters, and their respective values.

Conversion of metadata formats

In order to convert an XML file to JSON format, we need to extract the individual metadata parameters out of the ISO metadata XML file using the get_metadata_values function.

Replacing placeholders with actual values

The templates stored in the Excel spreadsheets contain ISO parameter placeholders. The product_dictionary2values function replaces these placeholders, saved in a product dictionary, with the actual metadata values.

Generating ISO metadata from product files

ISO metadata can be generated by extracting the necessary information based on the product files (using the generate_product_dictionary function). Depending on the product type, specific conversion functions are called:

• RTC products: gamma_rtc_metadata function

Building metadata structures

The ISO metadata is saved in two different formats: XML and JSON. The XML is using a tree structure that is built with the iso_xml_structure function, while the JSON structure is formed by a complex nested dictionary using the iso_dictionary_structure function.

Writing metadata to file

There are two separate functions to save the metadata to a file format. meta_xml_file is the function that generates an ISO XML metadata file, while meta_json_file saves the metadata in JSON format.

Cleaning up metadata

There are two separate functions to clean up the metadata structures. clean_xml_structure (for XML files) and clean_json_structure (for JSON files) is the function that removes metadata for product files (digital elevation model, incidence angle map, scattering area map) that are only optionally included.

ISO metadata tools

I developed a number of tools that perform the various tasks outline above.

Generate ISO template file from Excel spreadsheet

usage: generate_iso_template.py [-h] [-dem DEM] excelFile isoBase
Generate ISO template file from Excel spreadsheet
positional arguments:
excelFile   name of the Excel template spreadsheet
isoBase     basename of the ISO metadata template file
optional arguments:
-h, --help  show this help message and exit
-dem DEM    name of DEM template spreadsheet

Generate ISO metadata file from Excel spreadsheet

usage: generate_iso_metadata.py [-h] [-product PRODUCT] [-data DATA] [-dem DEM] meta_file log_file iso_base
Generate ISO metadata file from Excel spreadsheet
positional arguments:
meta_file         file path of product metadata such as a "manifest.safe" or leader file
log_file          name of RTC processing log file
iso_base          basename of the ISO XML metadata file
optional arguments:
-h, --help        show this help message and exit
-product PRODUCT  name of the product type (default: gamma_rtc)
choices: ["gamma_rtc"]
-data DATA        name of the data source (default: sentinel)
choices: ["sentinel"]
-dem DEM          DEM type used for terrain correction
choices: ["copernicus"]

At the moment, I have only added one function gamma_rtc_log2meta that analyzes GAMMA RTC log files for metadata. The corresponding product type is 'gamma_rtc'.

Author

Rudi Gens