Skip to main content

CWL runner for Kubernetes

Project description

calrissian

CWL on Kubernetes

CircleCI

Overview

Calrissian is a CWL implementation designed to run individual steps as Pods in a kubernetes cluster

It is in development and includes a simple workflow (revsort single / array) to reverse and sort the contents of text files.

Preparing Openshift

The openshift/ directory contains YAML files that demonstrate the basic functionality. To configure Openshift, do the following:

  1. Create a project in openshift (e.g. calrissian-demo-project). Note that namespaces must be unique, so if calrissian-demo-project is already in use on the cluster, you will need to choose a different name.

     oc new-project calrissian-demo-project
    
  2. Create a role in your project to allow managing Pods:

     oc create role pod-manager-role --verb=create,delete,list,watch --resource=pods
    
  3. Bind your project's default service account access to these roles. Calrissian running inside the cluster will use this service account to make Kubernetes API calls.

     oc create rolebinding pod-manager-default-binding --role=pod-manager-role --serviceaccount=calrissian-demo-project:default
    
  4. Create the BuildConfig - this allows Openshift to build the source code into a Docker image and starts a build automatically. If you wish to build a different branch or repo, edit this file.

     oc create -f openshift/BuildConfig.yaml
    
  5. Create the VolumeClaims - these are the storage locations that will be shared between pods, and must support read-write many access.

     oc create -f openshift/VolumeClaims.yaml
    

Running the example

With the VolumeClaims and Build processes in place, you can run the example with 2 jobs. These jobs use the docker image built by the BuildConfig, so they cannot be run until the build completes.

  1. Stage the input data and workflow onto the input-data volume with the StageInputDataJob.

     oc create -f openshift/StageInputDataJob.yaml
    
  2. Run the workflow engine with the CalrissianJob

     oc create -f openshift/CalrissianJob-revsort.yaml
    
  3. Job pods can be monitored from the openshift web interface or via command-line

     oc logs -f job/calrissian-revsort-array
    

Notes

  1. Calrissian will delete completed jobs for individual steps, but you may want to delete the calrissian-revsort-array job manually after running it.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distributions

No source distribution files available for this release.See tutorial on generating distribution archives.

Built Distributions

calrissian-0.3.1-py3.6.egg (41.8 kB view details)

Uploaded Egg

calrissian-0.3.1-py2.py3-none-any.whl (20.0 kB view details)

Uploaded Python 2Python 3

File details

Details for the file calrissian-0.3.1-py3.6.egg.

File metadata

  • Download URL: calrissian-0.3.1-py3.6.egg
  • Upload date:
  • Size: 41.8 kB
  • Tags: Egg
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.21.0 setuptools/40.6.2 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.6.8

File hashes

Hashes for calrissian-0.3.1-py3.6.egg
Algorithm Hash digest
SHA256 f5998739e6c145f0373704140e2370183ccc82caa14feda25024631b567a1a48
MD5 79d3acb308a56e378c4ab9000c177777
BLAKE2b-256 add94eef5aaed1defa68e3d96df7bf99c89acd467023bf26a387f6997663e778

See more details on using hashes here.

File details

Details for the file calrissian-0.3.1-py2.py3-none-any.whl.

File metadata

  • Download URL: calrissian-0.3.1-py2.py3-none-any.whl
  • Upload date:
  • Size: 20.0 kB
  • Tags: Python 2, Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/1.13.0 pkginfo/1.5.0.1 requests/2.21.0 setuptools/40.6.2 requests-toolbelt/0.9.1 tqdm/4.31.1 CPython/3.6.8

File hashes

Hashes for calrissian-0.3.1-py2.py3-none-any.whl
Algorithm Hash digest
SHA256 b01ed8793d0ac559c479dfa99fb3c3f58f73d853bc5c819db497d1601c7c1c5d
MD5 6bccd22a6ea16c9feb57628031fa85b7
BLAKE2b-256 811a414893370e16fbbf4ebf34297affff9f7434f928689dbdde7b67bc1b28e7

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page