python_workflow_submitter

Python alternative to creating and running argo workflows in the Data Analysis Platform

This allows for the submission of yaml files to Diamond's GraphQL API. This automatically lints yaml files before submission.

Notably, this requires a .env file in your src/ folder.

To submit your workflow notebook:

from python_workflow_submitter.submit_workflow import submit_workflow_yaml

await submit_workflow_yaml("example.yaml")

To submit a workflow script:

import asyncio
from python_workflow_submitter.submit_workflow import submit_workflow_yaml

asyncio.run(submit_workflow_yaml("example.yaml", visit="ks10000-3"))

To submit a generic workflow via a graphql mutation:

import asyncio
from python_workflow_submitter.submit_workflow import submit_workflow

asyncio.run(
    submit_workflow(
        "example-template",
        {"png": True, "jpg": False, "jpeg": True, "tif": True, "tiff": False},
        visit=str(os.environ.get("VISIT")),
    )
)

To list workflows:

import asyncio
from python_workflow_submitter.list_workflows import list_workflows

asyncio.run(list_workflows(limit=5, filter="EXAMPLES"))

To create a helm template in your helm chart folder

import asyncio
from python_workflow_submitter.create_helm_yaml import create_helm_yaml

create_helm_yaml(
    "notebook.yaml", "/workspaces/python_workflow_submitter/helm", "values/values.yaml"
)

To list workflows in specific visit with optional filtering:

import asyncio
from python_workflow_submitter.list_workflows import list_workflows_in_visit

asyncio.run(list_workflows(limit=5,filter=filter: {
    "creator": "gmg29649",
    "template": "example-template",
  	"workflowStatusFilter": {"succeeded": True}
  }))

To list information about a workflow in a visit with a specific name:

import asyncio
from python_workflow_submitter.list_workflows import list_workflows_in_visit

asyncio.run(info_about_workflow(name="conditional-steps-tswxm", visit="ks10000-3"))