deep-code 0.1.6

deepesdl earthcode integration utility tool

deep-code

deep-code is a lightweight python tool that comprises a command line interface(CLI) and Python API providing utilities that aid integration of DeepESDL datasets, experiments with EarthCODE.

The first release will focus on implementing the publish feature of DeepESDL experiments/workflow as OGC API record and Datasets as an OSC stac collection.

Setup

Install

deep-code will be available in PyPI for now and will be available in conda-forge in the near future. Till the stable release, developers/contributors can follow the below steps to install deep-code.

Installing from the repository for Developers/Contributors

To install deep-code directly from the git repository, clone the repository, and execute the steps below:

conda env create
conda activate deep-code
pip install -e .

This installs all the dependencies of deep-code into a fresh conda environment, and installs deep-code from the repository into the same environment.

Testing

To run the unit test suite:

pytest

To analyze test coverage

pytest --cov=deep-code

To produce an HTML coverage report

pytest --cov-report html --cov=deep-code

deep_code usage

deep_code provides a command-line tool called deep-code, which has several subcommands providing different utility functions. Use the --help option with these subcommands to get more details on usage.

The CLI retrieves the Git username and personal access token from a hidden file named .gitaccess. Ensure this file is located in the same directory where you execute the CLI command.

.gitaccess example

github-username: your-git-user
github-token: personal access token

deep-code generate-config

Generates starter configuration templates for publishing to EarthCODE openscience catalog.

Usage

deep-code generate-config [OPTIONS]

Options

 --output-dir, -o : Output directory (default: current)

Examples:

deep-code generate-config
deep-code generate-config -o ./configs

deep-code publish

Publishes metadata of experiment, workflow and dataset to the EarthCODE open-science catalog

Usage

deep-code publish DATASET_CONFIG WORKFLOW_CONFIG [--environment ENVIRONMENT] [--mode 
all|dataset|workflow]

Arguments

DATASET_CONFIG - Path to the dataset configuration YAML file
(e.g., dataset-config.yaml)

WORKFLOW_CONFIG - Path to the workflow configuration YAML file
(e.g., workflow-config.yaml)

Options

--dataset-config, - Explict path to dataset config
--workflow-config, - Explicit path to workflow config
--environment, -e - Target catalog environment:
production (default) | staging | testing
--mode, -m Publishing mode:
all (default) | dataset | workflow

Examples:

1. Publish to staging catalog

deep-code publish dataset-config.yaml workflow-config.yaml --environment=staging

2. Publish to testing catalog

deep-code publish dataset-config.yaml workflow-config.yaml -e testing

3. Publish to production catalog

deep-code publish dataset-config.yaml workflow-config.yaml

4. Publish Dataset only

deep-code publish dataset-config.yaml -m dataset

deep-code publish --dataset-config dataset.yaml -m dataset

5. Publish Workflow only

deep-code publish dataset-config.yaml -m workflow

deep-code publish --workflow-config workflow.yaml -m dataset

dataset-config.yaml example

dataset_id: esa-cci-permafrost-1x1151x1641-1.0.0.zarr
collection_id: esa-cci-permafrost
osc_themes:
  - cryosphere
osc_region: global
# non-mandatory
documentation_link: https://deepesdl.readthedocs.io/en/latest/datasets/esa-cci-permafrost-1x1151x1641-0-0-2-zarr
access_link: s3://deep-esdl-public/esa-cci-permafrost-1x1151x1641-1.0.0.zarr
dataset_status: completed

dataset-id has to be a valid dataset-id from deep-esdl-public s3 bucket or your team bucket.

workflow-config.yaml example

workflow_id: "esa-cci-permafrost"
properties:
  title: "ESA CCI permafrost"
  description: "cube generation workflow for esa-cci-permafrost"
  keywords:
    - Earth Science
  themes:
      - cryosphere
  license: proprietary
  jupyter_kernel_info:
    name: deepesdl-xcube-1.8.3
    python_version: 3.11
    env_file: "https://github.com/deepesdl/cube-gen/blob/main/Permafrost/environment.yml"
jupyter_notebook_url: "https://github.com/deepesdl/cube-gen/blob/main/Permafrost/Create-CCI-Permafrost-cube-EarthCODE.ipynb"
contact:
  - name: Tejas Morbagal Harish
    organization: Brockmann Consult GmbH
    links:
      - rel: "about"
        type: "text/html"
        href: "https://www.brockmann-consult.de/"