pelago

Helpers for compute-to-data algorithms on Ocean / Pontus-X: input files, consumer parameters and output files, plus reading sections of HDF5 files and writing PDF reports. The same script runs on your machine and on the node.

Install

pip install pelago                # Job and Param, standard library only
pip install "pelago[hdf5]"        # + H5File (h5py, numpy)
pip install "pelago[report]"      # + Report and plots (matplotlib, markdown, xhtml2pdf)
pip install "pelago[all]"         # everything, including pandas for H5File.frame()

Python 3.10 or newer.

Usage

from pelago import Job, Param

job = Job(
    Param.number("window", default=10000),
    Param.text("keyword", required=False),
    Param.select("mode", ["fast", "full"], default="fast"),
    Param.boolean("pdf", default=True),
)

for path in job.inputs:
    ...

job.write_json("summary.json", {"files": len(job.inputs), "window": job.params.window})

• job.inputs: all dataset files of the job, sorted by name. job.inputs_matching("*.csv"), job.input("*.csv"), job.require_inputs(n) and job.hdf5_inputs (HDF5 by extension or file signature) select from them.
• job.params: the parameter values, converted to the declared type. If the consumer did not set a value, the default is used. job.params.sources tells where each value came from.
• job.output(name), job.write_json, job.write_text, job.write_csv, job.save_figure: write to the output folder. Paths outside it are refused.
• job.log(msg) writes to the job log, job.fail(msg) stops the job with exit code 2.
• An invalid or missing required parameter stops the job with exit code 2 and one line in the log.

Start from examples/minimal.

HDF5

H5File reads the structure and attributes without touching the data, then only the rows and columns you ask for. It does not assume any layout; compound tables written by PyTables or pandas (format="table") work too.

from pelago import H5File

with H5File(path) as h5:
    print(h5.describe())                      # tree with shapes, types and fields
    h5.tree()                                 # {path: {"kind", "shape", "dtype", "fields", "attrs"}}
    h5.attrs("signals/acceleration")          # attributes; bytes decoded, JSON strings parsed
    part = h5.read("signals/acceleration", rows=slice(0, 10_000), columns=["time", "x"])
    for chunk in h5.chunks("signals/acceleration", size=100_000, columns=["x"]):
        ...
    df = h5.frame("signals/acceleration", rows=slice(0, 1000))    # pandas DataFrame
    h5.find("*image*")                        # paths matching a glob
    h5.save("pictures/img0", job.output("img0.png"))              # bytes or string dataset to a file

• read returns numpy arrays: a structured array for several fields of a compound dataset, a plain array for one field. For 2-D datasets columns are indexes of the second axis. Strings come back as str.
• save and raw handle bytes, strings, uint8 arrays and opaque values; data:...;base64, strings are decoded. index= picks one element of an array of images.
• A wrong path raises KeyError with the closest existing paths.

Reports

from pelago import Report
from pelago.plots import bar, line, heatmap

(Report("Summary")
    .text("## Result\n\nAll files passed.")          # Markdown
    .params(job.params)
    .table(rows, title="Files")                       # list of dicts, rows + columns, or a DataFrame
    .figure(line(t, {"x": x, "y": y}, xlabel="time [s]"), "Signal")
    .image("photo.png", "Setup")
    .save(job.output("report.pdf")))                  # .pdf via xhtml2pdf, anything else writes HTML

The style is built in. Change a few variables with Report(title, style={"font": "Times-Roman", "accent": "#b91c1c", "page": "A4 landscape"}) (also font_size, margin, image_width, colours ink, muted, rule, shade), or add CSS with Report(title, css="h1 { color: black; }"). .html() returns the whole page; figures and images are embedded. pelago.plots has bar, line and heatmap; each returns a matplotlib figure and needs no display.

Run locally

python main.py --inputs test-data --out outputs --param window=500 --param pdf=false
python main.py --describe        # consumerParameters for the algorithm's DDO

An algoCustomData.json in the input folder is read the same way as on the node. --params file.json and --param name=value override it, in that order. Without options the paths come from PELAGO_INPUTS and PELAGO_OUTPUTS (default /data/inputs, /data/outputs); --log or PELAGO_LOG set the log level.

How Ocean passes data to the job

Job also reads the per-DID layout (DIDS set, files in /data/inputs/<did>/0); job.inputs_by_did groups them. The node does not apply the parameter defaults from the DDO. If the consumer sets nothing, algoCustomData.json is {}. Some clients send numbers and booleans as strings. Job handles both.

Publish an algorithm

Bundle the script and pelago into one file and publish that file as the algorithm:

stickytape main.py --add-python-path <pelago>/src --output-file dist/algorithm.py
python dist/algorithm.py --describe > dist/consumer_parameters.json

Container: ttrau/datascience-base (pelago, numpy, h5py, matplotlib and the report libraries installed), entrypoint python $ALGO. Put the output of --describe into the algorithm's consumerParameters. examples/multi_asset_text has a build.sh that bundles an algorithm and tests it locally and in the container. examples/hdf5_sections reads sections of HDF5 inputs.

Development

python3 -m venv .venv && . .venv/bin/activate
make init install
make test

make clean removes build output, caches and example outputs.

Release

The version is set in one place: __version__ in src/pelago/__init__.py. Use semantic versions: raise the last number for fixes, the middle one for new features.

1. Set __version__, commit and push.
2. make release. It stops if the version is already on PyPI, if there are uncommitted changes, or if a test fails. Otherwise it builds, uploads to PyPI (credentials from ~/.pypirc or a PyPI token) and then pushes the tag v<version>.
3. Raise pelago>=<version> in the requirements of datascience-base-docker and build a new image.

make version prints the current version. If the upload worked but the tag push failed, push the tag by hand: git tag v<version> && git push origin v<version>.

License

MIT, see LICENSE.