pelagos-py 0.0.0

Customizable processing pipeline for oceanographic data from autonomous assets

Pelagos-py

Pelagos-py provides a flexible, modular pipeline framework for defining, executing, and visualising multi-step data-processing workflows for oceanographic data.

Each pipeline is composed of a series of steps that are automatically built from a central user-defined YAML configuration file. As Pelagos-py only depends on the config file to construct a pipeline, processing can be easily reproduced by others through sharing of configs.

Documentation

The documentation for this package is available here.

Please note that the documentation is still under construction.

Overview

The Pipeline class orchestrates the flow of data through a sequence of modular “steps.” Each step performs a specific processing task (e.g., data loading, quality control, profile detection, export).

Key characteristics:

Component	Description
Configuration-driven	Users define the workflow in a YAML file describing each step, its parameters, and diagnostics options.
In-build Quality Control	Pre-build quality control tests can be specified in the config by the user to flag bad data.
Diagnostics	Where possible, data is can be visualized to see the effect of each component of the pipeline.

Installation

Pelagos-py is not yet and installable package, so for the moment you have to make a local copy to run it:

git clone https://github.com/NOC-OBG-Autonomy/pelagos-py.git
cd pelagos-py
# create/activate a virtual environment
pip install -e . 

See Getting Started for more details.

How to run

1. Initialization

   Import the 'Pipeline' class and create a pipeline using your config (see below for example)

    from pelagos_py.pipeline import Pipeline
    pipeline = Pipeline(config_path="my_pipeline.yaml")
   

2. Pipeline Execution

   Running the pipeline executes each step defined by the config in order

    results = pipeline.run()
   

3. Diagnostics & Visualization

   Steps can optionally include diagnostic plots or summaries by setting:

    diagnostics: true
   

4. Exporting Pipeline Configuration

   The entire pipeline configuration can be exported to a YAML file for reproducibility:

   pipeline.export_config("exported_pipeline.yaml")
   

🧩 Example Configuration

An example YAML configuration for a simple pipeline. See examples/notebooks/pipeline_demo.ipynb for a full demo.

 # Pipeline Configuration
 pipeline:
   name: Example CTD Processing Pipeline
   description: A pipeline for processing CTD data
   visualisation: false
 
 steps:
   - name: Load OG1
     parameters:
       file_path: ../examples/data/OG1/Nelson_646_R.nc # Path to the input NetCDF file
     diagnostics: false
 
   - name: Derive CTD
     parameters:
       to_derive: [
         DEPTH,
         PRAC_SALINITY,
         ABS_SALINITY,
         CONS_TEMP,
         DENSITY
       ]
     diagnostics: false
 
   - name: "Data Export"
     parameters:
       export_format: "netcdf"
       output_path: "../examples/data/OG1/Nelson_646_R_Processed.nc"

🔁 Extending the Pipeline

A full breakdown can be found here: Developer Guide.

License

Apache 2.0 License

Copyright 2025-2026 The National Oceanography Centre and contributors.