libica 1.1.0

Python SDK for Illumina Connected Analytics

libica

Python SDK to programmatically call Illumina Connected Analytics (ICA) BioInformatics web services i.e. SDK for API https://ica-docs.readme.io/reference

• Workflow Execution Service (WES)
• Task Execution Service (TES)
• Genomic Data Store (GDS)
• Developer Console Service (DCS)
• Event Notification Service (ENS)

Overview:

• https://umccr-illumina.github.io/libica/
• Tested for Python 3.6, 3.7, 3.8, 3.9
• Test Coverage

Using OpenAPI Package

• Install through pip like so:

pip install libica

• Export ICA base URL and JWT Bearer token:

export ICA_BASE_URL=<baseUrl>
export ICA_ACCESS_TOKEN=<tok>

• Generate Bearer JWT token using ICA CLI like so:

ica tokens create --help

• Somewhere in your Python code:

import os
from libica.openapi import libwes
from libica.openapi.libwes import WorkflowList, WorkflowCompact

ica_base_url = os.getenv("ICA_BASE_URL")
ica_access_token = os.getenv("ICA_ACCESS_TOKEN")

configuration = libwes.Configuration(
    host=ica_base_url,
    api_key={
        'Authorization': ica_access_token
    },
    api_key_prefix={
        'Authorization': "Bearer"
    },
)

with libwes.ApiClient(configuration) as api_client:

    wfl_api: libwes.WorkflowsApi = libwes.WorkflowsApi(api_client)
    try:
        page_token = None
        while True:
            wfl_list: WorkflowList = wfl_api.list_workflows(page_size=100, page_token=page_token)
            # print(wfl_list)
            for item in wfl_list.items:
                wfl: WorkflowCompact = item
                print(wfl.id)
                print(wfl.name)
            page_token = wfl_list.next_page_token
            if not wfl_list.next_page_token:
                break
    except libwes.ApiException as e:
        print(e)

More examples available at:

• libica.openapi documentation
• PyDoc
• Wiki

Using App Package

NOTE: libica.app package contains reusable modules that are based on use cases around UMCCR Data Portal backend, CWL Workflows and its orchestration implementations. Hence, it may be a specific implementation, however, it can still be reusable for your cases. Please have a look into package and quick starter examples are as follows.

Example: Configuration Object Builder

from libica.app import configuration
from libica.openapi import libgds

gds_config = configuration(
  lib=libgds,  # pass in library of interest e.g. libwes, libtes, etc 
  secret_name=["FROM_AWS_SECRET_MANAGER_THAT_STORE_ICA_ACCESS_TOKEN"],
  base_url="https://use1.platform.illumina.com",  # overwrite if not https://aps2.platform.illumina.com
  debug=False,  # True if you like to debug API calls, False by default anyway, just for demo
)

with libgds.ApiClient(gds_config) as api_client:
    ...

Example: Listing Files from GDS

from typing import List

from libica.app import gds
from libica.openapi import libgds

vol, path = gds.parse_path("gds://development/some/folder/path/")
files: List[libgds.FileResponse] = gds.get_file_list(volume_name=vol, path=path)

for file in files:
  print(f"{file.name}, {file.volume_name}, {file.path}, {file.presigned_url}")

Development

• Setup virtual environment and activate it

• Install dependencies

make install

• To bring up mock API μ-services

make up

• To run tests suites

make unit
make autounit

• To run full suite, smoke test

make test

AutoGen Workflow

• SDK is autogenerated from respective endpoint OpenAPI (Swagger) definitions
• There are few key CLI tools for this autogen workflow to work.
  1. openapi-generator-cli -- used to generate code and doc
  2. swagger-cli -- validate definitions
  3. prism-cli -- setup mock integration test
• These CLI tools are Node.js app, hence, required build tools node, npm, npx and yarn as follows.

node -v
 v14.17.3
npm -i -g yarn
yarn install
npx openapi-generator-cli help
npx swagger-cli --help
npx prism --help

• However, prism mock server is set up through docker compose as follows.

make up
make test_ica_mock
docker-compose logs
docker-compose ps
docker ps

• All the autogen config and arrangement refer to syncapi.sh which is called through Makefile targets.

License

MIT License and DISCLAIMER