l2o-corealg 0.1.0

ESA ROSE-L Level-2 Ocean Processor Core Algorithm Library

L2O-COREALG

This repository contains the ROSE-L Level-2 Ocean Processor Core Algorithm Library, L2O-COREALG.

Project Structure

This project is organized as follows:

• docs contains the source of the documentation.
• src contains the source code of the project's main Python package.
• tests contains the unit and integration tests of the project.

Tools

The following tools are used by this project:

• .gitlab-ci.yml: GitLab CI pipeline including the following tools:

  Click to expand the list

  • ruff: Python code linter and formatter
  • mypy: Python static type checker
  • xenon: Python code complexity scanner
  • docstr-coverage: Python docstring coverage checker
  • bandit: Python code security scanner
  • pip-audit: Python package dependency vulnerability scanner
  • pytest: Python unit & integration tests
  • build: Python packaging frontend
  • setuptools: Python packaging backend
  • twine: Python package uploader
  • sphinx: Documentation generation

• pre-commit-config.yaml: Pre-commit hooks executed on commits

• pyproject.toml: Build system requirements for Python packaging

• environment.yml: Package list for setting up a development environment using Conda

• requirements.txt: Package list for setting up a development environment using PIP

• .vscode: Settings and recommendations for the VS Code IDE

• docs/conf.py: Sphinx documentation generator configuration

Documentation

The documentation is available online at: https://rl2ocean.pages.gitlab.dlr.de/l2o-corealg

Installation

Assuming an active Python 3.13 environment, the package can be installed as follows:

pip install .

Contributing

This section provides detailed instructions for setting up a complete development environment for developing and testing the Python module.

Note: The following steps have been tested on Linux operating systems (Debian- and RHEL-based).

Step 1: Clone repository

Obtain a local copy of the repository by cloning it using:

git clone https://gitlab.dlr.de/rl2ocean/l2o-corealg.git
cd l2o-corealg

Step 2: Set up development environment

Working with the code requires installation of Python version 3.13 and various dependencies. The easiest way to achieve this is by setting up an environment, which can be done in different ways. Instructions are provided below for:

1. Conda environment using Miniforge (recommended)
2. Python virtual environment using venv

Conda is recommended as the easier option because it provides a full environment including Python itself and non-Python dependencies.

Depending on your choice, expand and continue with the respective section below, but not both.

a. Conda using Miniforge

Download Miniforge and follow the installation instructions: https://github.com/conda-forge/miniforge

If not done during the installation, initialize Conda for your shell:

~/miniforge3/bin/conda init

Create the environment using the provided environment.yml file. If you want to install the environment to a directory different from the default (~/miniforge3/envs/), you can specify it using the --prefix option as shown:

conda env create -f environment.yml
-or-
conda env create -f environment.yml --prefix /path/to/rl2ocean_env

Finally, activate the environment:

conda activate rl2ocean_env
-or-
conda activate /path/to/rl2ocean_env

In case the environment.yml file was updated by other developers, the existing Conda environment can be updated. Make sure the environment is activated and execute:

conda env update -f environment.yml --prune

b. Python venv

Python and the non-Python dependencies need to be installed using the operating system's package manager. For Debian-based or RHEL-based distributions, respectively, run:

sudo apt install python3.13 libnetcdf-c++4-dev libgdal-dev
-or-
sudo dnf install python3.13 netcdf-cxx4-devel gdal-devel

If your operating system's repositories do not offer these dependencies, especially Python 3.13, they have to be installed in another way. This is not covered here.

To create a virtual environment, execute in the l2o-corealg directory:

python3.13 -m venv .venv

Activate the virtual environment:

source .venv/bin/activate

Step 3: Install package in editable mode

After setting up the development environment using one of the methods explained above, the l2o-corealg package has to be installed in editable mode.

The following line will automatically install the dependencies for development and l2o-corealg itself using the pyproject.toml file:

pip install -e ".[dev]"

Step 4: Install pre-commit hooks

Pre-commit hooks help catch coding issues locally before pushing to the online repository. This is achieved by running a set of tools automatically whenever a Git commit is triggered.

A selection of pre-commit hooks are configured for the repository. For details, see the configuration file .pre-commit-config.yaml.

To install the pre-commit hooks, activate the development environment and execute:

pre-commit install

From now on, your code will be checked for formatting and more whenever you commit changes using Git. See the console output for more information, especially if your code gets rejected.

Step 5: Run tests

Testing the code is crucial to ensure correct functionality and avoid regressions. The pytest framework is used to define and execute a set of unit and integration tests. They are stored in the tests directory.

To execute the tests locally using pytest, run:

python -m pytest tests/

This is also a good way of ensuring that setting up the development environment was successful.

The GitLab CI pipeline will automatically run the unit and integration tests as well.

Step 6: Set up integrated development environment (IDE) Visual Studio Code (VS Code)

This section describes how to set up and use the VS Code IDE for development. It is by no means exhaustive, and of course you may use a different IDE altogether if you prefer.

Using an IDE such as VS Code allows you to:

• Navigate through the repository using the GUI.
• Develop Python source code supported by syntax highlighting, type checking, auto-completion, automated formatting and more.
• Debug the Python module utilizing the graphical debugging view.
• Use the in-built Git support to manage branches, view diffs side-by-side, commit and push your changes to Gitlab.

For this the .vscode folder contains the following files:

• extensions.json: Recommended extensions for working with the code
• settings.json: Workspace settings for the recommended extensions
• launch.json: Basic launch configuration for running the module with the Python debugger

Note: The following steps assume that the VS Code IDE is running on the same machine as the code itself, but VS Code can also be used for remote development using the Remote - SSH and Remote - WSL extensions. See this website for further information: https://code.visualstudio.com/docs/remote/remote-overview (for using Remote - SSH more project specific steps can be opened after step 1 below)

Follow the steps of the first-time set up in order:

1. Download and install VS Code for your platform. Remember to disable telemetry.

2. Remote - SSH based setup (optional)

   Install “Remote SSH” extension (search for “ssh” in extensions on left side)

   Press "><" button in lower left corner “Open a Remote Window”

   Press “Connect to Host” in top center field

   Type host name or IP (or user name plus host name/IP, e.g. [e-mail] or user_name@123.456.789.123) and press enter, a new VS Code window should open

   Select OS (if applicable, accept SHA fingerprint key)

   Enter your credentials

   On lower left, “details” of connection status can be opened. After pressing “details” password might have to be entered again a few times

   Now you should be able to work on the remote machine as if working locally, following the next step, but all paths (e.g. source folder, interpreter) are on the remote machine

3. Open the root directory of this repository (l2o-corealg) in VS Code.

4. A pop-up will appear asking whether you trust the authors. Click on Yes, I trust the authors, otherwise the files will be read-only.

5. You will be prompted to install the recommended workspace extensions, which you should confirm. If you missed that or changes to extensions.json were committed, you can also navigate to "Extensions", filter extensions using "Recommended" filter and than install/uninstall recommended/not recommended extensions.

6. Point the IDE to the Python environment created above by selecting the correct Python interpreter:

   1. Bring up the command palette with Ctrl+Shift+P.
   2. Type >Python: Select Interpreter and hit Enter.
   3. Select the Python 3.13.X executable from your Conda or virtual environment from the list. If it does not show up, type in the (relative or absolute) path manually.

7. If everything worked the Python module imports in .py files should now resolve.

A note on debugging: The repository contains a very basic debug configuration inside the .vscode/launch.json file which runs the l2o-corealg module as an executable (python -m l2o). This can be run by hitting the F5 key, which should automatically switch to the debug view. For more information on Python debugging with VS Code, see: https://code.visualstudio.com/docs/python/debugging

Optional: Packaging

Developers are not expected to package the module themselves. The GitLab CI pipeline will automatically build the package and publish it to the package registry on each tag. Nevertheless, it can be useful to test locally whether packaging works.

To build the Python module of the project using build, run:

python -m build .

Branching, Committing, Requesting for Merge

Direct commits to the main branch are disabled. Please create your own branch for working on the source code, to which you can then iteratively commit your developments. When your branch work is completed, please execute

mypy -p l2o.corealg

to check type annotations and fix all detected problems. When all problems are solved you can initiate a merge request. Without mypy execution before the merge request the CI/CD pipeline might fail and your merge request will be denied.

Contributing to thematic processing, preprocessing and postprocessing

The orchestration of thematic processing (wind, wave, currents) needs to be implemented in the respective classes Wind, Wave or Current of the l2o.vam sub-package, while the actual core functions constituting the thematic processing need to be implemented in l2o.corealg, i.e. l2o.corealg.wind, l2o.corealg.wave and l2o.corealg.current. The value adding module for wind processing and the core functions corresponding to wind processing can be used as an example for getting started or for creating a new thematic processor module (first working commit: https://gitlab.dlr.de/rl2ocean/l2o-gpp/-/tree/f34c6eb8387b7041ee28bfb30924f21cfb5a47a2).

Analogically, contributions to pre- and postprocessing can be implemented in the classes Preprocessing or Postprocessing and in l2o.corealg.preprocessing or l2o.corealg.postprocessing, respectively.

When you want to create new thematic processor module, please inherit from the abstract class ValueAdding in l2o.vam to enable interfacing with the higher level processor and ensure compatibility.

Documentation generation

The project's documentation can be generated using Sphinx. The GitLab CI pipeline will generate and deploy the documentation when run for the default branch. It will generate the documentation accessible through Gitlab Pages from the project's docs folder.

The generation of the API documentation from the Python docstrings included in the source code is included in the documentation generation process.