Skip to main content

Python library for dataflow programming with Amazon SWF

Project description

Simpleflow

PyPI - Version GitHub Actions Workflow Status

Simpleflow is a Python library that provides abstractions to write programs in the distributed dataflow paradigm. It coordinates the execution of distributed tasks with Amazon SWF.

It relies on futures to describe the dependencies between tasks. A Future object models the asynchronous execution of a computation that may end. It tries to mimic the interface of the Python concurrent.futures library.

Features

  • Provides a Future abstraction to define dependencies between tasks.
  • Define asynchronous tasks from callables.
  • Handle workflows with Amazon SWF.
  • Implement replay behavior like the Amazon Flow framework.
  • Handle retry of tasks that failed.
  • Automatically register decorated tasks.
  • Encodes/decodes large fields to S3 objects transparently (aka "jumbo fields").
  • Handle the completion of a decision with more than 100 tasks.
  • Provides a local executor to check a workflow without Amazon SWF (see simpleflow --local command).
  • Provides decider and activity worker process for execution with Amazon SWF.
  • Ships with the simpleflow command. simpleflow --help for more information about the commands it supports.

You can read more in the Features section of the documentation.

Overview

Please read and even run the demo script to have a quick glance of simpleflow commands. To run the demo you will need to start decider and activity worker processes.

Start a decider with:

$ PYTHONPATH=$PWD simpleflow decider.start --domain TestDomain --task-list test examples.basic.BasicWorkflow

Start an activity worker with:

$ simpleflow worker.start --domain TestDomain --task-list quickstart

Then execute ./extras/demo.

More information

Read the main documentation at https://botify-labs.github.io/simpleflow/.

Project details


Release history Release notifications | RSS feed

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

simpleflow-0.34.1.tar.gz (113.0 kB view details)

Uploaded Source

Built Distribution

simpleflow-0.34.1-py3-none-any.whl (157.7 kB view details)

Uploaded Python 3

File details

Details for the file simpleflow-0.34.1.tar.gz.

File metadata

  • Download URL: simpleflow-0.34.1.tar.gz
  • Upload date:
  • Size: 113.0 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: uv/0.6.14

File hashes

Hashes for simpleflow-0.34.1.tar.gz
Algorithm Hash digest
SHA256 dd86fe90b658dd846f6103be166eb097fc758eb777b68c80adc7387a0eb478b1
MD5 4753a354d23a2781caa0bbcb23200ff1
BLAKE2b-256 19b74e8c72c454e7a0b137162b6d8e95616de44fa0a2f622adb916b323ffdec8

See more details on using hashes here.

File details

Details for the file simpleflow-0.34.1-py3-none-any.whl.

File metadata

File hashes

Hashes for simpleflow-0.34.1-py3-none-any.whl
Algorithm Hash digest
SHA256 32085ff52c2fd88b6a697ae49c50652988f57a28b1fad87b3198cac50d214211
MD5 b47ab97c1263073cc9509e5810f377d4
BLAKE2b-256 2c5967a9fd0572bf9e273c6c015fb4134c3a1133a82359a71311e0be2694fb45

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page