python-fastflow 0.8.1

A workflow engine based on Kopf

Fastflow

Deploy to Kubernetes with helm chart

kubectl create ns fastflow
helm -n fastflow upgrade --install fastflow chart

To install in more namespaces, create the namespace and run the helm command again, this time with the --skip-crds flag.

kubectl create ns fastflow-dev
helm -n fastflow-dev upgrade --install fastflow chart --skip-crds

Run examples

Apply examples

kubectl -n fastflow create -f examples/01-helloworld/workflow.yaml
kubectl -n fastflow create -f examples/02-digraph/workflow.yaml

Inspect results

kubectl -n fastflow get workflows
kubectl -n fastflow get tasks -l workflow=helloworld

Developer setup

For developing the fastflow project

Generate CRDS

python3 generate_crds.py

Create CRDS

Can also be installed by applying the helm chart

kubectl create -f chart/crds/kopfpeering-crd.yaml
find chart/crds/generated -name *.yaml -exec kubectl create -f '{}' \;

Delete CRDS

find chart/crds/generated -name *.yaml -exec kubectl delete -f '{}' \;

Virtual environment

python3 -m venv ~/venvs/fastflow
. ~/venvs/fastflow/bin/activate
python3 -m pip install --upgrade pip
python3 -m pip install -e .

Run from outside cluster

Will use kubectl config for cluster access. Greate for development, and can run with debugger attached.

Prepare namespace

kubectl create ns fastflow-dev
kubectl -n fastflow-dev apply -f - << EOYML
apiVersion: kopf.dev/v1
kind: KopfPeering
metadata:
  name: default
EOYML

Run as module (useful for debugger)

python3 -m fastflow \
--namespace fastflow-dev \
--dev

Run from cli

fastflow --namespace fastflow-dev --dev

Run Tests

Install test-requirements

python3 -m pip install -e . -r test-requirements.txt

Prepare namespace

kubectl create ns fastflow-test
kubectl -n fastflow-test apply -f test/kopf-test-peering.yaml

Run the tests as module

python3 -m pytest --color=yes

Run the tests from cli

pytest --color=yes

Building whl package and Docker image

Cleanup old packages

rm -Rf dist

Build package

python3 -m pip install build
python3 -m build

Build Docker image using minikube

rm -Rf dist && python3 -m build
eval $(minikube -p minikube docker-env)
DOCKER_BUILDKIT=1 docker build -t fastflow .

Use helm to run the image in Kubernetes, by specifying the image tag we just created

helm -n fastflow-dev upgrade --install --set imageOverride=fastflow fastflow chart

Example using custom Task implementations

The Operator needs to be able to load the custom code. Here is an example using the examples/03-farmlife example workflow. This workflow uses custom tasks implemented in examples/03-farmlife/tasks-impl/farmlife.py.

Run the operator locally

Run the operator so it can load the custom code

fastflow --dev --namespace fastflow-dev examples/03-farmlife/tasks-impl/farmlife.py

Apply the example workflow

kubectl -n fastflow-dev create -f examples/03-farmlife/workflow.yaml