Skip to main content

Asynchronous and distributed scheduling of Ewoks workflows from python

Project description

ewoksjob

Utilities for job scheduling of ewoks workflows.

Ewoksjob provides an ewoks interface for asynchronous and distributed scheduling of ewoks from python.

Note that ewoksjob distributes the execution of workflows while ewoksdask distributes the execution of tasks in a workflow. So in the context of workflows, job scheduling exists on two levels.

The primary clients that need to schedule workflows are

Installation

Install on the client side

pip install ewoksjob

Install on the worker side

pip install ewoksjob[worker]

Getting started

Start a worker pool that can execute ewoks graphs

ewoksjob worker

Start a workflow on the client side

from ewoksjob.client import submit

workflow = {"graph": {"id": "mygraph"}}
future = submit(args=(workflow,))
result = future.get()

Note that both environments need to be able to import celeryconfig which contains celery configuration (mainly the message broker and result backend URL's).

Hello world example

Clone the git repository and start a worker pool

scripts/worker.sh --sql

Submit workflows

scripts/runjobs.sh --sql

Tests

pytest --pyargs ewoksjob

To run the redis tests you need redis-server (e.g. conda install redis-server).

Documentation

https://ewoksjob.readthedocs.io/

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

ewoksjob-1.1.0.tar.gz (28.8 kB view details)

Uploaded Source

File details

Details for the file ewoksjob-1.1.0.tar.gz.

File metadata

  • Download URL: ewoksjob-1.1.0.tar.gz
  • Upload date:
  • Size: 28.8 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: twine/6.1.0 CPython/3.12.10

File hashes

Hashes for ewoksjob-1.1.0.tar.gz
Algorithm Hash digest
SHA256 cad957c40ca2e186aec092b5f69a95b365020b7782c438f115db1a7353afd54e
MD5 462728c6b237540a05da7b487c3921ca
BLAKE2b-256 586488c8ecd874db8ad5c3da3e04d85b333124273d747b1b7a5770fa85dddc12

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page