cowait 0.5.0

Cowait is a framework for creating containerized workflows with asynchronous Python.

cowait

Cowait is a framework for creating containerized distributed applications with asynchronous Python. Containerized functions, called Tasks, can run locally in Docker or on remote Kubernetes clusters. The intention is to build a novel type of workflow engine that offers maximum flexibility while also requiring minimal setup and configuration. Because of the loose definition of what a Task could be, and the integration with Docker & Kubernetes, Cowait can be used to easily build many types of distributed applications.

Check out the documentation at https://cowait.io/docs/quick-start/

Notice

Cowait is still in fairly early development. We invite you to try it out and gladly accept your feedback and contributions, but please be aware that breaking API changes may be introduced.

Community

Join the Cowait development Slack channel!

Get your invite at https://slack.cowait.io

Getting Started

Requirements

• docker
• python 3.6+

Installation

$ pip install cowait

Task Development

First Task

1. Write a task.

Tasks can be written as decorated functions or subclasses of cowait.Task. The easiest way is to write a decorated function:

# hello.py
from cowait import task

@task
async def Hello():
    print('Hello World')

2. Build a task image

All files within the current directory is bundled into a docker image.

In the same folder as hello.py, run:

$ cowait build

3. Run it locally

To run a task, pass the name of the module import name to cowait run. Since our task code lives in hello.py, the module name will be hello. In the same folder as hello.py, run:

$ cowait run hello

Notes

• hello supplied to cowait run is the python module name. This module should contain exactly one task class. Modules can be single python files or subdirectories with __init__.py files.
• The actual function/class name of the task does not matter when running from the CLI, only when importing and executing tasks from python.

Subtasks, Arguments & Return Values

Tasks can execute subtasks by importing and calling asynchronously using the await keyword. Subtasks accept named arguments and return results, much like regular functions.

Suppose we have a Square task that squares an integer.

# square.py
from cowait import task

@task
async def Square(number: int) -> int:
    return number ** 2

We can import it to another task and execute it:

# main.py
from cowait import task
from square import Square

@task
async def MainTask():
    value = 22
    squared = await Square(number=22)
    print(f'{value} squared is {squared}!')

Task Contexts

A task context is a directory containing a cowait.yml configuration file. This directory will act as the root of a project, and everything within this folder is copied into the resulting docker image during the build step. If you have not created a cowait.yml file, the current working directory (when executing cowait build) will be used.

** Example Task Context **

 /var/stuff/my_cowait_project
   src/
     library.py
   cowait.yml
   requirements.txt
   task1.py
   task2.py

In this case, /var/stuff/my_cowait_project will be the context directory.

Pushing Task Images

Before you can run tasks on a Kubernetes cluster, they must be pushed to a docker registry that is accessible from the cluster. To do this, you must define a proper image name in the context configuration file.

version: 1
cowait:
  image: docker.io/username/cowait-task

Once the full image name has been configured, you can push the image using cowait push:

$ cowait push

Cluster Deployment

Requiurements

• Configured kubernetes cluster context.
• Docker registry accessible from the cluster.

First, ensure that the image has been pushed to your docker registry.

$ cowait push

Kubernetes Configuration

Before you can run tasks on a kubernetes cluster, you need a service account with permissions to create/list pods. A sample configuration is provided in k8setup.yml, which will allow any pods owned by the default service account to create, list and destroy other pods.

For detailed information about running on Kubernetes, see the documentation.

Running Tasks

To run tasks on a remote cluster, pass the cluster name using the --cluster option to cowait run.

Cowait comes with a predefined cluster called kubernetes which always refers to your default configured cluster (the one returned by $ kubectl config current-context).

$ cowait run docker.io/username/cowait-task --cluster kubernetes

Development

Requirements

• docker
• python 3.6+
• poetry (recommended)

Installation

1. Clone repository
2. $ python3 -m pip install -e .

Changes to the cowait/ directory require a rebuild of the base image. You can do this with the provided helper script in the root of the repository:

$ ./build.sh

Note: Tasks will have to rebuilt with cowait build for the changes to take effect.

Generate Documentation You can generate documentation (from docstrings) for Cowait to the /docs folder using pdoc3 (defined in Poetry as a dev dependency):

$ pdoc3 --skip-errors --html --output-dir docs/html cowait