oceanprotocol-job-details 0.4.5

A Python package to get details from OceanProtocol jobs

OceanProtocol Job Details

A Python package to get details from OceanProtocol jobs

Installation

pip install oceanprotocol-job-details
#or
uv add oceanprotocol-job-details

Usage

As a simple library, we only need to import load_job_details and run it. It will:

1. Read from disk the needed parameters to populate the JobDetails from the given base_dir. Looking for the files corresponding to the passed DIDs in the filesystem according to the Ocean Protocol Structure.
2. If given a InputParameters type that inherits from pydantic.BaseModel, it will create an instance from the environment variables.

Minimal Example

from oceanprotocol_job_details import load_job_details

job_details = load_job_details({"base_dir": "...", "transformation_did": "..."})

Custom Input Parameters

If our algorithm has custom input parameters and we want to load them into our algorithm, we can do it as follows:

from pydantic import BaseModel
from oceanprotocol_job_details import load_job_details


class Foo(BaseModel):
    bar: str


class InputParameters(BaseModel):
    # Allows for nested types
    foo: Foo


job_details = load_job_details({"base_dir": "...", "transformation_did": "..."}, InputParameters)

# Usage
parameters = await job_details.input_parameters()
parameters.foo
parameters.foo.bar

The values to fill the custom InputParameters will be parsed from the algoCustomData.json located next to the input data directories.

Iterating Input Files the clean way

from oceanprotocol_job_details import load_job_details


job_details = load_job_details(...)

for idx, file_path in job_details.inputs():
    ...

_, file_path = next(job_details.inputs())

OceanProtocol Structure

data        # Root /data directory
├── ddos    # Contains the loaded dataset's DDO (metadata)
│   ├── 17feb...e42 # DDO file
│   └── ... # One DDO per loaded dataset
├── inputs  # Datasets dir
│   ├── 17feb...e42 # Dir holding the data of its name DID, contains files named 0..X
│   │   └── 0 # Data file
│   └── algoCustomData.json # Custom algorithm input data
├── logs    # Algorithm output logs dir
└── outputs # Algorithm output files dir

Note: Even though it's possible that the algorithm is passed multiple datasets, right now the implementation only allows to use one dataset per algorithm execution, so normally the executing job will only have one ddo, one dir inside inputs, and one data file named 0.