coloc_sat

coloc_sat is a Python package for co-locating satellite data products. It allows you to co-locate data from different satellite sources based on provided paths and common variable names. This README provides an installation guide and instructions for usage. This package also allows co-location listings. Input satellites / missions that can be treated by this tool are the following : WindSat / SMOS / SMAP / SAR (L1/L2) / ERA5 / HY2 / ASCAT. SAR satellites are RCM, RadarSat-2 and Sentinel1.

Installation

Make sure you have Python 3.9 or higher installed.

Using pip

xsar is a dependency of coloc_sat that depends on GDAL. To avoid conflicts during the installation of coloc_sat, gdal must be installed beforehand using conda.

conda install -c conda-forge gdal
pip install coloc-sat

Using conda

conda install -c conda-forge coloc_sat

Using mamba

mamba install -c conda-forge coloc_sat

Additionnaly, to use RCM data, xarray-safe-rcm must be installed (not yet available on conda-forge)

pip install xarray-safe-rcm

Usage

Configuration

Before using coloc_sat, you need to configure the paths to your satellite data products and optionnaly define common variable names. Follow the steps below:

1. Create a directory named coloc_sat in your home directory.
2. Inside the coloc_sat directory, create a file named localconfig.yml.

In localconfig.yml, fill in the paths to your satellite products following the schema below:

paths:
  SMOS:
    - '/path/to/SMOS/%Y/%(dayOfYear)/*%Y%m%d*.nc'
    - '/path2/to/SMOS//%Y/%(dayOfYear)/*%Y%m%d*.nc'
  HY2:
    - '/path/to/hy2/%Y/%(dayOfYear)/*%Y%m%d*.nc'
  ASCAT:
    - '/path/to/ascat/%Y/%(dayOfYear)/*%Y%m%d*.nc'
  ERA5:
    - '/path/to/era5/%Y/%m/era_5-copernicus__%Y%m%d.nc'
  RS2:
    L1:
      - '/path/to/rs2/L1/*/%Y/%(dayOfYear)/RS2*%Y%m%d*'
    L2:
      - '/path/to/rs2/L2/*/%Y/%(dayOfYear)/RS2_OK*/RS2_*%Y%m%d*/post_processing/nclight_L2M/rs2*owi*%Y%m%d*0003*_ll_gd.nc'
  S1:
    L1:
      - '/path/to/s1/L1/*/*/%Y/%(dayOfYear)/S1*%Y%m%d*SAFE'
    L2:
      - '/path/to/s1/L2/*/%Y/%(dayOfYear)/S1*%Y%m%d*/post_processing/nclight_L2M/s1*owi*%Y%m%d*000003*_ll_gd.nc'
      - '/path2/to/s1/L2/*/%Y/%(dayOfYear)/S1*%Y%m%d*/post_processing/nclight_L2M/s1*owi*%Y%m%d*0003*_ll_gd.nc'
  RCM:
    L1:
      - '/path/to/rcm/L1/*/%Y/%(dayOfYear)/RCM*%Y%m%d*'
    L2: []
  WS:
    - '/path/to/windsat/%Y/%(dayOfYear)/wsat_%Y%m%d*.gz'
  SMAP:
    - '/path/to/smap/%Y/%(dayOfYear)/RSS_smap_*.nc'
    - '/path2/to/smap/%Y/%(dayOfYear)/RSS_smap_*.nc'

load_variables:
  "Soil Moisture and Ocean Salinity":
    - wind_speed
    - measurement_time
  "RADARSAT Constellation 2":
    - wind_speed
    - wind_from_direction
    - time

common_var_names:
  wind_speed: wind_speed
  wind_direction: wind_direction_ecmwf
  wind_from_direction: wind_from_direction
  longitude: lon
  latitude: lat
  time: time

Replace the paths with the actual paths to your satellite data products. Use the placeholders %Y, %m, %d, and %(dayOfYear) to automatically parse dates from the paths. Only the path for the products you want to co-located with are needed.
load_variables can be used if a subset of products variables must be loaded, instead of all by default.
See config_explained.yaml for some more details.

Co-locating Data

Once you've configured the paths and common variable names, you can use coloc_sat to co-locate the data. Import the package and start co-locating your data based on your needs.

Now, import the package:

import coloc_sat

Then, define important variables for the co-location:

delta_time=60
destination_folder = '/tmp'
listing = True
product_generation = True
product1 = '/path/to/s1/l2/s1a-ew-owi-cm-20181009t142906-20181009t143110-000003-02A122_ll_gd.nc'

Example code for co-locating a satellite product with a mission:

ds_name = 'SMOS'
# Call the generation tool
generator = coloc_sat.GenerateColoc(product1_id=product1, ds_name=ds_namedelta_time=delta_time, product_generation=product_generation, 
                            listing=listing, destination_folder=destination_folder)
# save the results (listing and / or co-location products)
generator.save_results()

NOTE : It is also possible to use this co-location generator with the console. Here are examples.

a) This first example shows how to generate a coloc between 2 specified products:

Coloc_2_products --product1_id /path/to/rs2/L2/rs2--owi-cm-20141004t210600-20141004t210715-00003-BDBE0_ll_gd.nc --product2_id path/to/s1/L2/s1a-iw-owi-cm-20141004t211657-20141004t211829-000003-002FF5_ll_gd.nc --listing --product_generation

b) This second example shows how to generate all possible coloc between a product and a mission (all products from this mission):

Coloc_between_product_and_mission --product1_id /path/to/rs2/L2/rs2--owi-cm-20141004t210600-20141004t210715-00003-BDBE0_ll_gd.nc --mission_name S1 --listing --product_generation

Example of resulting listing of co-located products

Default parameters for the listing filename is 'listing_coloc_' + 'MISSION_NAME1' + '_' + 'MISSION_NAME2' + '_' + 'delta_time' + '.txt'

Example of product_name : 'listing_coloc_ERA5_SAR_60.txt'

Note : For RCM, RadarSat-2 and RCM, 'SAR' is used.

Content:

/path/to/era5/era_5-copernicus__20181009.nc:path/to/S1/L2/s1a-ew-owi-cm-20181009t142906-20181009t143110-000003-02A122_ll_gd.nc

Example of resulting xarray co-location product

Default parameters for the co-location product filename is 'sat_coloc_' + 'product_name1' + '__' + 'product_name2' + '.nc'

Example of product name: 'sat_coloc_s1a-ew-owi-cm-20181009t142906-20181009t143110-000003-02A122_ll_gd__era_5-copernicus__20181009.nc'

<xarray.Dataset>
    Dimensions:                            (lat: 14, lon: 9)
    Coordinates:
      * lon                                (lon) float32 -131.0 -130.5 ... -127.0
      * lat                                (lat) float32 13.5 14.0 ... 19.5 20.0
        time                               datetime64[ns] ...
        spatial_ref                        int64 ...
    Data variables: (12/52)
        wind_streaks_orientation_stddev_1  (lat, lon) float32 ...
        elevation_angle_1                  (lat, lon) float32 ...
        heading_angle_1                    (lat, lon) float32 ...
        nesz_cross_corrected_1             (lat, lon) float32 ...
        nrcs_co_1                          (lat, lon) float32 ...
        mask_flag_1                        (lat, lon) float32 ...
        ...                                 ...
        mwd_2                              (lat, lon) float32 ...
        tcw_2                              (lat, lon) float64 ...
        mwp_2                              (lat, lon) float32 ...
        tp_2                               (lat, lon) float64 ...
        mdww_2                             (lat, lon) float32 ...
        mpww_2                             (lat, lon) float32 ...
    Attributes: (12/28)
        Conventions_1:           CF-1.6
        title_1:                 SAR ocean surface wind field
        institution_1:           IFREMER/CLS
        reference_1:             Mouche Alexis, Chapron Bertrand, Knaff John, Zha...
        measurementDate_1:       2018-10-09T14:30:08Z
        sourceProduct_1:         s1a-ew-owi-cm-20181009t142906-20181009t143110-00...
        ...                      ...
        footprint_2:             POLYGON ((-131 13.5, -131 20, -127 20, -127 13.5...
        counted_points:          0
        vmax_m_s:                nan
        Bias:                    0
        Standard deviation:      0
        scatter_index:           nan

🛈 When co-locating swath-to-swath products (e.g., SAR with ASCAT or SAR with HY2), the resulting co-located product is gridded on a 2D latitude/longitude grid, where the lat/lon values vary along both dimensions. This is due to the geophysical swath shape of the instruments, which is preserved during the co-location process.

<xarray.Dataset> Size: 8kB
  Dimensions:                      (y: 19, x: 2)
  Coordinates:
      lon                          (y, x) float64 304B ...
      lat                          (y, x) float64 304B ...
  Dimensions without coordinates: y, x
  Data variables: (12/25)
      wind_speed_1                 (y, x) float64 304B ...
      wind_speed_co_1              (y, x) float64 304B ...
      wind_speed_cross_1           (y, x) float64 304B ...
      wind_from_direction_1        (y, x) float64 304B ...
      wind_from_direction_co_1     (y, x) float64 304B ...
      wind_from_direction_cross_1  (y, x) float64 304B ...
      ...                           ...
      owiMask_Nrcs_1               (y, x) float64 304B ...
      owiMask_Nrcs_cross_1         (y, x) float64 304B ...
      time_1                       (y, x) datetime64[ns] 304B ...
      wind_direction_2             (y, x) float64 304B ...
      wind_speed_2                 (y, x) float64 304B ...
      time_2                       (y, x) datetime64[ns] 304B ...
  Attributes: (12/18)
      sourceProduct_1:         s1a-ew-owi-cm-20210924t210331-20210924t210556-00...
      missionName_1:           SENTINEL-1 A
      measurementStartDate_1:  2021-09-24T21:04:43.000000000
      measurementStopDate_1:   2021-09-24T21:04:43.000000000
      footprint_1:             POLYGON ((-44.47028963435961 6.859109024551801, ...
      time_difference:         3577000000000 nanoseconds
      ...                      ...
      footprint_2:             POLYGON ((-42.67162999999999 14.30404, -42.67162...
      counted_points:          22
      vmax_m_s:                19.868717277486912
      Bias:                    0.17677608494815503
      Standard deviation:      0.5309783623632102
      scatter_index:           6.406290982900541

Important notes

This library is a Work-in-progress, so that some acquisition type combinations aren't treated yet:

Acknowledgements

Special thanks to REMSS for their Windsat reader.

• Free software: MIT license
• Documentation: https://coloc-sat.readthedocs.io.