Skip to main content

Outflow is a framework that helps you create and execute sequential, parallel as well as distributed task workflows.

Reason this release was yanked:

old release

Project description

Outflow is a framework that helps you build and run task workflows.

The api is as simple as possible while still giving the user full control over the definition and execution of the workflows.

Feature highlight :

  • Simple but powerful API
  • Support for parallelized and distributed execution
  • Centralized command line interface for your pipeline commands
  • Integrated database access, sqlalchemy models and alembic migrations
  • Executions and exceptions logging for tracability
  • Strict type and input/output checking for a robust pipeline

Check out our documentation for more information.

Installing

Install and update using pip:

pip install -U outflow

Quick start

One file starter

Create a pipeline.py script:

# -- pipeline.py
from outflow.core.commands import Command, RootCommand
from outflow.core.pipeline import Pipeline
from outflow.core.tasks import Task


# with the as_task decorator, the function will be automatically converted into a Task subclass
# the signature of the function, including the return type, is used to determine task inputs and outputs
@Task.as_task
def GetValues() -> {'a': str, 'b': str}:
    return {'a': 'hello', 'b': 'world'}

# default values can also be used as inputs
@Task.as_task
def PrintValues(a: str, b: str, c: str = '?' ):
    print(f"{a} {b}{c}")

@RootCommand.subcommand()
class HelloWorld(Command):

    def setup_tasks(self):
        # instantiate tasks
        get_values = GetValues()

        # you can specify inputs value during instantiation
        print_values = PrintValues(c="!")

        # build the workflow
        get_values >> print_values

        # return the terminating task(s) of the workflow
        # they will be used as entrypoints to navigate through the execution tree
        return [print_values]

if __name__ == "__main__":
    # instantiate and run the pipeline
    with Pipeline(
        root_directory=None,
        settings_module="outflow.core.pipeline.default_settings",
        force_dry_run=True,
    ) as pipeline:
        result =  pipeline.run()

and run your first Outflow pipeline:

$ python pipeline.py hello_world

A robust, configurable and well-organized pipeline

You had a brief overview of Outflow's features and you want to go further. Outflow offers command line tools to help you to start your pipeline project.

First, we will need to auto-generate the pipeline structure -- a collection of files including the pipeline settings, the database and the cluster configuration, etc.

$ python -m outflow management create pipeline my_pipeline

Then, we have to create a plugin -- a dedicated folder regrouping the commands, the tasks as well as the description of the database (the models)

$ python -m outflow management create plugin my_namespace.my_plugin --plugin_dir my_pipeline/plugins/my_plugin

In the my_pipeline/settings.py file, add your new plugin to the plugin list:

PLUGINS = [
    'outflow.management',
    'my_namespace.my_plugin',
]

and run the following command:

$ python ./my_pipeline/manage.py my_plugin

You'll see the following output on the command line:

 * outflow.core.pipeline.pipeline - pipeline.py:325 - INFO - No cluster config found in configuration file, running in a local cluster
 * my_namespace.my_plugin.commands - commands.py:49 - INFO - Hello from my_plugin

Your pipeline is up and running. You can now start adding new tasks and commands.

Contributing

For guidance on setting up a development environment and how to make a contribution to Outflow, see the contributing guidelines.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

outflow-0.3.1.tar.gz (71.6 kB view details)

Uploaded Source

Built Distribution

outflow-0.3.1-py3-none-any.whl (102.4 kB view details)

Uploaded Python 3

File details

Details for the file outflow-0.3.1.tar.gz.

File metadata

  • Download URL: outflow-0.3.1.tar.gz
  • Upload date:
  • Size: 71.6 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.1.6 CPython/3.8.9 Linux/5.11.16-arch1-1

File hashes

Hashes for outflow-0.3.1.tar.gz
Algorithm Hash digest
SHA256 2c7e9c9093cdc8827fcfd232816bd804bcd6cb44a522320d3330979948bcf5ec
MD5 4019dd3185a3ff8240a62dba09fe9e29
BLAKE2b-256 1965e7558ead46578613dca0731ee7f95368f7e496a58e696e718a07b5747a13

See more details on using hashes here.

File details

Details for the file outflow-0.3.1-py3-none-any.whl.

File metadata

  • Download URL: outflow-0.3.1-py3-none-any.whl
  • Upload date:
  • Size: 102.4 kB
  • Tags: Python 3
  • Uploaded using Trusted Publishing? No
  • Uploaded via: poetry/1.1.6 CPython/3.8.9 Linux/5.11.16-arch1-1

File hashes

Hashes for outflow-0.3.1-py3-none-any.whl
Algorithm Hash digest
SHA256 3812c51b83171d01638eedbbc4bbf1d1332e6bd5b50ca0077220be4b3636c51b
MD5 4634af220da1625e26c050e6b7917ef2
BLAKE2b-256 913b1c87d53f4550b4033b0ff5cc8e575d366dc9a64b914148e22052b99b6e9f

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page