im-data-manager-job-decoder 2.0.0

Job decoding logic

Informatics Matters Data Manager Job Decoder

A package that simplifies the decoding of encoded text strings. Given an encoded string the decode() method returns the decoded value or an error.

For example, given the following jinja2 encoded string '{{ foo }}, bar={{ bar }}, baz={{ baz }}' and variable map {'foo': 1, 'bar': 2, 'baz': 3} the decoder returns the string 'foo=1, bar=2, baz=3'.

The following encoding/decoding formats are supported: -

• jinja2 (3.0)

The package also provides validate_job_schema() and validate_manifest_schema() functions, which can (should) be used to validate the Manifests and Job definitions against the built-in schemas.

Installation (Python)

The Job decoder is published on PyPI and can be installed from there:

pip install im-data-manager-job-decoder

Once installed you can validate the definition (expected to be a dictionary formed from the definition YAML file) with:

>>> from decoder import decoder
>>> error: Optional[str] = decoder.validate_manifest_schema(manifest)
>>> error: Optional[str] = decoder.validate_job_schema(job_definition)

And you can decode encoded fields within the job definition. Here we’re decoding the job’s ‘command’ using a map of variables and their values:

>>> decoded, success = decoder.decode(definition['command'],
                                      variables,
                                      'command',
                                      decoder.TextEncoding.JINJA2_3_0)

A method also exists to generate a Data Manger Job documentation URL for the supported repository types. For example, to get the auto-generated documentation URL for a job definition hosted in a GitHub repository you can do this:

>>> doc_url: str = decoder.get_job_doc_url("github",
                                           collection
                                           job_id,
                                           job_definition,
                                           manifest_url)

We support github and gitlab as repository types.

Get in touch

• Report bugs, suggest features or view the source code on GitHub.