labcas.workflow 0.1.9

Get Planetary Data from the Planetary Data System (PDS)

LabCas Workflow

Run workflows for Labcas

Depending on what you do, there are multiple ways of running a labcase workflow:

• Developers: for developers: local run, natively running on your OS
• Integrators: for AWS Managed Apache Airflow integrators (mwaa), with a local mwaa
• System Administrators: for System administors, deployed/configured on AWS
• End users: For end users, using the AWS deployment.

Developers

The tasks of the workflow run independently from Airflow. TODO: integrate to the airflow python API.

Install

With python 3.11, preferably use a virtual environment

pip install -e '.[dev]'

Set AWS connection

./aws-login.darwin.amd64
export AWS_PROFILE=saml-pub

Run/Test the client

python src/labcas/workflow/manager/main.py

Deploy package on pypi

Upgrade the version in file "src/labcas/workflow/VERSION.txt"

Publish the package on pypi:

pip install build
pip install twine
rm dist/*
python -m build
twine upload dist/*

Integrators

Build the Dask worker image

Update the labcas.workflow dependency version as needed in docker/Dockerfile, then:

docker build -f docker/Dockerfile . -t labcas/workflow

Create a managed AirFlow docker image to be run locally

Use repository https://github.com/aws/aws-mwaa-local-runner, clone it, then:

./mwaa-local-env build-image

Then from your local labcas_workflow repository:

cd mwaa

As needed, update requirements in requirements directory and dags in dags directory.

Update the AWS credentials

aws-login.darwin.amd64
cp -r ~/.aws .

Launch the server

docker compose -f docker-compose-local.yml up

Test the server on http://localhost:8080 , login admin/test

Stop

Ctrl^C

Stop and re-initialize local volumes

docker compose  -f ./docker/docker-compose-local.yml down -v

See the console on http://localhost:8080, admin/test

Test the requirement.txt files

./mwaa-local-env test-requirements

Debug the workflow import

docker container ls

Pick the container id of image "amazon/mwaa-local:2_10_3", for example '54706271b7fc':

Then open a bash interpreter in the docker container:

docker exec -it 54706271b7fc bash

And, in the bash prompt:

cd dags
python3 -c "import nebraska"

Start the scheduler:

docker network create dask
docker run --network dask -p 8787:8787 -p 8786:8786 labcas/workflow scheduler

Start one worker

docker run  --network dask -p 8786:8786 labcas/workflow worker 

Start the client, same as in following section

With dask on ECS

Deploy the image created in the previous section on ECR

Have a s3 bucket labcas-infra for the terraform state.

Other pre-requisites are:

• a VPC
• subnets
• a security group allowing incoming request whre the client runs, at JPL, on EC2 or Airflow, to port 8786 and port 8787
• a task role allowing to write on CloudWatch
• a task execution role which pull image from ECR and standard ECS task Excecution role policy "AmazonECSTaskExecutionRolePolicy"

Deploy the ECS cluster with the following terraform command:

cd terraform
terraform init
terraform apply \
    -var consortium="edrn" \
    -var venue="dev" \
    -var aws_fg_image=<uri of the docker image deployed on ECR>
    -var aws_fg_subnets=<private subnets of the AWS account> \
    -var aws_fg_vpc=<vpc of the AWS account> \
    -var aws_fg_security_groups  <security group> \
    -var ecs_task_role <arn of a task role>
    -var ecs_task_execution_role <arn of task execution role>

Run

Set you local AWS credentials to access the data

./aws-login.darwin.amd64
export AWS_PROFILE=saml-pub

Start the dask cluster

Run the processing

python ./src/labcas/workflow/manager/main.py

Publish the package on pypi

pip install build
pip install twine
python -m build
twine upload dist/*

Apache Airflow