This repository contains a library for the generation an OGC complaint application package from a Jupyter Notebook by parsing its contents and metadata. 

This library serves as a base upon which to build project specific application generation software. For example the Unity Project uses this libary for their Unity specific application genetation software.

mdps-app-generator | unity-example-application

Features

• Extracts information from application Git repositorties for metadata reporting
• Creates an algorithm descriptor file that can be used by an OGC compliant ADES server
• Creates Common Workflow Language (CWL) files that expose the notebook's arguments
• Creates a Docker image called by the CWL file to execute parsed notebook applications

Install from PyPi

pip install ogc-ipynb-app-gen

Preparing Your repository

Any application repository used by this library must:

1. Contain an valid Jupyter notebook which can be run using papermill
2. Contain a valid configuration file for repo2docker (generally an environment.yml or requirements.txt)

NOTE: papermill is required to listed as a dependencies in your configuration file.

By default repo2docker will a certain version of Python, currently Python 3.8. Be sure to provied a runtime.txt configuration file to specify a different version if your packages require it.

Preparing Your Notebook

In your Jupyter notebook file, the input parameters will be automatically determined from a code cell in the notebook annotated with the parameters tag. An attempt is made to automatically detect the type of the parameters. When that fails, type can be specified through a type hint in the form of a comment following the variable in the form # type: <type_name>. See below for examples:

example_argument_int = 1
example_argument_float = 1.0
example_argument_string = "string"
example_argument_bool = True
example_argument_empty = None # type: string Allow a null value or a string

Not that a comment can still be provided but it must be given after the type name. If a value is specified with the None it will be come an Any type inside of the generated CWL unless a type hint is provided. Parameters like the above are passed to the generated CWL through a parameters block in the CWL job input file. For example, the following YAML would be part of the job input file for the above parameters:

parameters:
    example_argument_string: "string"
    example_argument_int: 1
    example_argument_float: 1.0
    example_argument_empty: "Not null string"
    example_argument_bool: True

If any of the values in the CWL job input file are given as null then the default value inside the notebook will be used.

Two special type hints are used to connect inputs and outputs into the generated CWL file from the ADES processing system. These hints are given by the stage-in and stage-out type hints:

input_stac_collection_file = 'test/stage_in/stage_in_results.json' # type: stage-in
output_stac_catalog_dir    = 'process_results/'                    # type: stage-out

The stage-in variable will be passed a filename to either a STAC Collection file or a STAC Catalog file. The data staged for the application will be located in the same directory as the file. It is recommended to use the Unity-Py Collection.from_stac method as illustrated in the unity-example-application. This method will automatically detect the type of file being passed and handle them both seemlessly.

The stage-out variable will be passed path to the location where a STAC Catalog file should be written along with any output data. Application results must be located in this directory for them to be properly staged. Again, please follow the example in the unity-example-application archive for using Unity-Py to write such a catalog.

If either or both stage-in and stage-out variables are omitted from the notebook then the related CWL file will not be produced.

License

See our: LICENSE