Skip to main content
Join the official 2019 Python Developers SurveyStart the survey!

CWL runner for Kubernetes

Project description

Calrissian

CWL on Kubernetes

CircleCI

Overview

Calrissian is a CWL implementation designed to run inside a Kubernetes cluster. Its goal is to be highly efficient and scalable, taking advantage of high capacity clusters to run many steps in parallel.

Cluster Requirements

Calrissian requires a Kubernetes or Openshift/OKD cluster, configured to provision PersistentVolumes with the ReadWriteMany access mode. Kubernetes installers and cloud providers don't usually include this type of storage, so it may require additional configuration.

Calrissian has been tested with NFS using the nfs-client-provisioner and with GlusterFS using OKD Containerized GlusterFS. Many cloud providers have an NFS offering, which integrates easily using the nfs-client-provisioner.

Scalability / Resource Requirements

Calrissian is designed to issue tasks in parallel if they are independent, and thanks to Kubernetes, should be able to run very large parallel workloads.

When running calrissian, you must provide a limit the the number of CPU cores (--max-cores) and RAM megabytes (--max-ram) to use concurrently. Calrissian will use CWL ResourceRequirements to track usage and stay within the limits provided. We highly recommend using accurate ResourceRequirements in your workloads, so that they can be scheduled efficiently and are less likely to be terminated or refused by the cluster.

CWL Conformance

Calrissian leverages cwltool heavily and most conformance tests for CWL v1.0. Please see conformance for further details and processes.

To view open issues related to conformance, see the conformance label on the issue tracker.

Setup

Please see examples for installation and setup instructions.

Environment Variables

Calrissian's behaviors can be customized by setting the following environment variables in the container specification.

Pod lifecycle

By default, pods for a job step will be deleted after termination

  • CALRISSIAN_DELETE_PODS: Default true. If false, job step pods will not be deleted.

Kubernetes API retries

When encountering a Kubernetes API exception, Calrissian uses a library to retry API calls with an exponential backoff. See the tenacity documentation for details.

  • RETRY_MULTIPLIER: Default 5. Unit for multiplying the exponent interval.
  • RETRY_MIN: Default 5. Minimum interval between retries.
  • RETRY_MAX: Default 1200. Maximum interval between retries.
  • RETRY_ATTEMPTS: Default 10. Max number of retries before giving up.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for calrissian, version 0.9.0
Filename, size File type Python version Upload date Hashes
Filename, size calrissian-0.9.0-py2.py3-none-any.whl (28.4 kB) File type Wheel Python version py2.py3 Upload date Hashes View hashes
Filename, size calrissian-0.9.0-py3.7.egg (60.0 kB) File type Egg Python version 3.7 Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page