copernicus-marine-client 0.8.16

No project description provided

Copernicus Marine Service client

Features

The copernicus-marine-client offers capabilities through both the Command Line and Python interfaces:

• Metadata Information: List and retrieve metadata information on all variables, datasets, products, and their associated documentation.
• Subset Datasets: Subset datasets to extract only the parts of interest, in preferred format, such as Analysis-Ready Cloud-Optimized (ARCO) Zarr or NetCDF file format.
• Advanced Filters: Apply simple or advanced filters to get multiple files, in original formats like NetCDF/GeoTIFF, via direct Marine Data Store connections.
• No Quotas: Enjoy no quotas, neither on volume size nor bandwidth.

Installation

For installation, multiple options are available depending on your setup:

Conda|Mamba

No conda package has been created yet. Meanwhile, these steps cover the installation in a new isolated environment (safer). Replace conda by mamba if necessary.

• Create the file copernicus-marine-client-env.yml that contains:

name: cmc
channels:
  - conda-forge
dependencies:
  - pip
  - pip:
    - copernicus-marine-client

• Use the terminal or an [Conda|Mamba] Prompt to create the cmc environment from the yml file:

conda env create --file copernicus-marine-client-env.yml

• Open the new cmc environment by running:

conda activate cmc

• Verify that the new cmc environment was installed correctly:

conda env list

Pip

Otherwise, if you already have an environment (safer to clone it), the package can be installed using the pip command:

python -m pip install copernicus-marine-client

And to upgrade the package to the newest available version, run:

python -m pip install copernicus-marine-client --upgrade

User Guide

For more comprehensive details on how to use the copernicus-marine-client and make the most out of the next-generation data services, please refer to our Help Center. It ensures a smooth migration for existing users of legacy services such as MOTU, OPeNDAP, and FTP.

Getting Started

The --help option

To discover commands and their available options, consider appending --help on any command line.

Example:

copernicus-marine --help

Returns:

Usage: copernicus-marine [OPTIONS] COMMAND [ARGS]...

Options:
  -V, --version  Show the version and exit.
  --help         Show this message and exit.

Commands:
  describe  Print Copernicus Marine catalog as JSON
  login     This command check the copernicus-marine credentials provided...
  get       Download originally produced data files
  subset    Downloads subsets of datasets as NetCDF files or Zarr stores

Command describe

Retrieve information about all products and datasets as JSON:

copernicus-marine describe --include-datasets

The JSON output can also be saved like follows:

copernicus-marine describe --include-datasets > all_datasets_copernicus-marine.json

Command login

Create a single configuration file .copernicus_marine_client_credentials allowing to access all Marine Data Store data services. By default, it saves file in user's home directory.

Example:

> copernicus-marine login
username : johndoe
password :
INFO - root - Configuration files stored in /Users/foo/.copernicus_marine_client

If .copernicus_marine_client_credentials already exists, the user is asked for confirmation to overwrite (--overwrite/--overwrite-configuration-file).

Access points migration and evolution

If you already have a configuration for current services (e.g. ~/motuclient/motuclient-python.ini, ~/.netrc or ~/_netrc) in your home directory, it will automatically be taken into account with commands get and subset without the need for running the login command. If the configuration files are already available in another directory, when running commands subset or get, you can use the --configuration-file-directory option to specify the path.

Command subset

Remotely subset a dataset, based on variable names, geographical and temporal parameters.

Example:

copernicus-marine subset --dataset-id cmems_mod_ibi_phy_my_0.083deg-3D_P1Y-m --variable thetao --variable so --start-datetime 2021-01-01 --end-datetime 2021-01-03 --minimal-longitude 0.0 --maximal-longitude 0.1 --minimal-latitude 28.0 --maximal-latitude 28.1

Returns:

INFO     - root - Download through S3
<xarray.Dataset>
Dimensions:    (depth: 50, latitude: 2, longitude: 1, time: 1)
Coordinates:
  * depth      (depth) float32 0.5058 1.556 2.668 ... 5.292e+03 5.698e+03
  * latitude   (latitude) float32 28.0 28.08
  * longitude  (longitude) float32 0.08333
  * time       (time) datetime64[ns] 2021-01-01
Data variables:
    thetao     (time, depth, latitude, longitude) float32 dask.array<chunksize=(1, 50, 2, 1), meta=np.ndarray>
    so         (time, depth, latitude, longitude) float32 dask.array<chunksize=(1, 50, 2, 1), meta=np.ndarray>
Attributes: (12/19)
    Conventions:    CF-1.0
    bulletin_date:  2022-11-01
    ...             ...
    title:          CMEMS IBI REANALYSIS: YEARLY PHYSICAL PRODUCTS
Do you want to proceed with download? [Y/n]:

By default, after the display of summary dataset subset, a download confirmation is asked. To force the download and skip this user's action, call option --force-download.

Note about longitude range

Options --minimal-longitude FLOAT and --maximal-longitude FLOAT work as follows:

• If maximal-longitude - minimal-longitude >= 360, return the full dataset.
• Else:
  • If the requested range does not cross the antemeridian, the result dataset will be between -180° and 180°.
  • If it does cross the antemeridian, the result dataset will be between 0° and 360°.

Note that you can request any longitudes you want. A modulus is applied to bring the result between -180° and 360°. For example, if you request [530°, 560°], the result dataset will be in [170°, 200°].

Access point migration and evolution

The copernicus marine client will download the data in the most efficient way according to your request:

• if the target dataset is available in ARCO version, then files are downloaded in a fresh new data.zarr folder in the current working directory.
• if the target dataset is not yet available in ARCO version, then a data.nc file is downloaded in the current working directory.

However, the output directory and filename can be specified using -o/--output-directory and -f/--output-filename respectively.

If the name ends with .nc, it will be written as a NetCDF file.

You can force the use of a specific data access service with option --force-service.

Command get

Download the dataset file(s) as originally produced, based on the datasetID or the path to files.

Example:

copernicus-marine get --dataset-url ftp://my.cmems-du.eu/Core/IBI_MULTIYEAR_PHY_005_002/cmems_mod_ibi_phy_my_0.083deg-3D_P1D-m/2000/09

Returns:

INFO     - root - Downloading using service ftp...
INFO     - root - You requested the download of the following files:
Core/IBI_MULTIYEAR_PHY_005_002/cmems_mod_ibi_phy_my_0.083deg-3D_P1D-m/2000/09/CMEMS_v5r1_IBI_PHY_MY_PdE_01dav_20000901_20000901_R20201201_RE01.nc - 13.22 MB
[... truncated for brevity...]
Core/IBI_MULTIYEAR_PHY_005_002/cmems_mod_ibi_phy_my_0.083deg-3D_P1D-m/2000/09/CMEMS_v5r1_IBI_PHY_MY_PdE_01dav_20000920_20000920_R20201201_RE01.nc - 13.20 MB
Printed 20 out of 30 files

Total size of the download: 395.64 MB
Do you want to proceed with download? [Y/n]:

By default:

• after the header display with a summary of the request, a download confirmation is asked. To force the download and skip this user's action, add option --force-download.
• files are downloaded to the current directory applying the original folder structure but destination can be specified with the --output-directory option. Furthermore, if coupled with --no-directories, it will not recreate the original folder structure.

Option --show-outputnames displays the full paths of the output files, if required.

Note about filtering options

Option --filter allows to specify a Unix shell-style wildcard pattern (see fnmatch — Unix filename pattern matching) and select specific files:

copernicus-marine get --dataset-id cmems_mod_ibi_phy_my_0.083deg-3D_P1Y-m --filter "*01yav_200[0-2]*"

Returns:

INFO     - root - Downloading using service files...
INFO     - root - You requested the download of the following files:
s3://mdl-native/native/IBI_MULTIYEAR_PHY_005_002/cmems_mod_ibi_phy_my_0.083deg-3D_P1Y-m_202211/CMEMS_v5r1_IBI_PHY_MY_NL_01yav_20000101_20001231_R20221101_RE01.nc - 8.93 MB
s3://mdl-native/native/IBI_MULTIYEAR_PHY_005_002/cmems_mod_ibi_phy_my_0.083deg-3D_P1Y-m_202211/CMEMS_v5r1_IBI_PHY_MY_NL_01yav_20010101_20011231_R20221101_RE01.nc - 8.91 MB
s3://mdl-native/native/IBI_MULTIYEAR_PHY_005_002/cmems_mod_ibi_phy_my_0.083deg-3D_P1Y-m_202211/CMEMS_v5r1_IBI_PHY_MY_NL_01yav_20020101_20021231_R20221101_RE01.nc - 8.75 MB

Total size of the download: 26.59 MB
Do you want to proceed with download? [Y/n]:

Option --regex allows to specify a regular expression for more advanced files selection:

copernicus-marine get -i cmems_mod_ibi_phy_my_0.083deg-3D_P1Y-m --regex ".*01yav_20(00|01|02).*.nc"

Returns:

INFO     - root - Downloading using service files...
INFO     - root - You requested the download of the following files:
s3://mdl-native/native/IBI_MULTIYEAR_PHY_005_002/cmems_mod_ibi_phy_my_0.083deg-3D_P1Y-m_202211/CMEMS_v5r1_IBI_PHY_MY_NL_01yav_20000101_20001231_R20221101_RE01.nc - 8.93 MB
s3://mdl-native/native/IBI_MULTIYEAR_PHY_005_002/cmems_mod_ibi_phy_my_0.083deg-3D_P1Y-m_202211/CMEMS_v5r1_IBI_PHY_MY_NL_01yav_20010101_20011231_R20221101_RE01.nc - 8.91 MB
s3://mdl-native/native/IBI_MULTIYEAR_PHY_005_002/cmems_mod_ibi_phy_my_0.083deg-3D_P1Y-m_202211/CMEMS_v5r1_IBI_PHY_MY_NL_01yav_20020101_20021231_R20221101_RE01.nc - 8.75 MB

Total size of the download: 26.59 MB
Do you want to proceed with download? [Y/n]:

Shared options

Both subset and get commands provide these options:

Option --overwrite-output-data

When specified, the existing files will be overwritten. Otherwise, if the files already exist on destination, new ones with a unique index will be created once the download has been accepted (or once --force-download is provided).

Option --request-file

This option allows to specify CLI options but in a provided JSON file, quite useful for batch processing.

• Template for subset data request:

{
	"dataset_id": "cmems_mod_glo_phy-thetao_anfc_0.083deg_P1D-m",
	"start_datetime": "2022-04-11",
	"end_datetime": "2023-08-11",
	"minimal_longitude": -182.79,
	"maximal_longitude": -179.69,
	"minimal_latitude": -40,
	"maximal_latitude": -36,
	"minimal_depth": 0,
	"maximal_depth": 0,
	"variables": ["thetao"],
	"output_directory": "./data/",
	"output_filename": "temperature_small_pacific_2022208-202308.zarr",
	"force_download": false
}

Example:

copernicus-marine subset --request-file template_subset_data_request.json

• Template for get data request:

{
    "dataset_id": "cmems_mod_ibi_phy_my_0.083deg-3D_P1Y-m",
    "filter": "*01yav_200[0-2]*",
    "force_download": false,
    "force_service": "files",
    "log_level": "INFO",
    "no_directories": false,
    "no_metadata_cache": false,
    "output_directory": "./data/",
    "overwrite_output_data": false,
    "overwrite_metadata_cache": false,
    "show_outputnames": true,
}

Example:

copernicus-marine get --request-file template_get_data_request.json

Documentation

See the Help Center. A detailed API documentation will come at a later stage.

Contribution

We welcome contributions from the community to enhance this package. If you find any issues or have suggestions for improvements, please check out our Contribution Guidelines.

Future improvements

• Make available the currently not managed Analysis-Ready Cloud-Optimized (ARCO) versions of Ocean Monitoring Indicator (OMI), in situ and static datasets.
• Allow to specify the compression level when downloading your subset as NetCDF file.
• Allow to subset variables using their standard_name(s) and not only their name(s).

Roadmap

To discover the next milestones, please check out our Roadmap.

Join the community

Get in touch!

• Create your Copernicus Marine Account
• Chat with us (bottom right corner of Copernicus Marine Service)
• Join our training workshops
• (re)Tweet y/our Copernicus Stories
• Watch our videos