schedula 0.3.0

Produce a plan that dispatches calls based on a graph of functions, satisfying data dependencies.

.. _start-intro:


What is schedula?
*****************

Schedula implements a intelligent function scheduler, which selects
and executes functions. The order (workflow) is calculated from the
provided inputs and the requested outputs. A function is executed when
all its dependencies (i.e., inputs, input domain) are satisfied and
when at least one of its outputs has to be calculated.

Note: Schedula is performing the runtime selection of the
**minimum-workflow** to be invoked. A workflow describes the
overall process - i.e., the order of function execution - and it is
defined by a directed acyclic graph (DAG). The **minimum-workflow**
is the DAG where each output is calculated using the shortest path
from the provided inputs. The path is calculated on the basis of a
weighed directed graph (data-flow diagram) with a modified Dijkstra
algorithm.


Installation
************

To install it use (with root privileges):

::

$ pip install schedula

Or download the last git version and use (with root privileges):

::

$ python setup.py install


Install extras
==============

Some additional functionality is enabled installing the following
extras:

* plot: enables the plot of the Dispatcher model and workflow (see
``plot()``).

* web: enables to build a dispatcher Flask app (see ``web()``).

* sphinx: enables the sphinx extension directives (i.e., autosummary
and dispatcher).

* parallel: enables the parallel execution of Dispatcher model.

To install schedula and all extras, do:

::

$ pip install schedula[all]

Note: ``plot`` extra requires to install Graphviz (`download page
<https://www.graphviz.org/download/>`_). Make sure that the
directory containing the ``dot`` executable is on your systems’
path.

.. _end-quick:


Why may I use schedula?
***********************

Imagine we have a system of interdependent functions - i.e. the inputs
of a function are the output for one or more function(s), and we do
not know which input the user will provide and which output will
request. With a normal scheduler you would have to code all possible
implementations. I’m bored to think and code all possible combinations
of inputs and outputs from a model.


Solution
========

Schedula allows to write a simple model (``Dispatcher``) with just the
basic functions, then the ``Dispatcher`` will select and execute the
proper functions for the given inputs and the requested outputs.
Moreover, schedula provides a flexible framework for structuring code.
It allows to extract sub-models from a bigger one and to run your
model asynchronously or in parallel without extra coding.

Note: A successful `application <https://github.com/JRCSTU/CO2MPAS-TA>`_
is CO_2MPAS, where schedula has been used to model an entire
`vehicle <https://co2mpas.io/explanation.html#execution-model>`_.


Very simple example
*******************

Let’s assume that we have to extract some filesystem attributes and we
do not know which inputs the user will provide. The code below shows
how to create a ``Dispatcher`` adding the functions that define your
system. Note that with this simple system the maximum number of inputs
combinations is 31 ((2^n - 1), where *n* is the number of data).

..

>>> import schedula as sh
>>> import os.path as osp
>>> dsp = sh.Dispatcher()
>>> dsp.add_data(data_id='dirname', default_value='.', initial_dist=2)
'dirname'
>>> dsp.add_function(function=osp.split, inputs=['path'],
... outputs=['dirname', 'basename'])
'split'
>>> dsp.add_function(function=osp.splitext, inputs=['basename'],
... outputs=['fname', 'suffix'])
'splitext'
>>> dsp.add_function(function=osp.join, inputs=['dirname', 'basename'],
... outputs=['path'])
'join'
>>> dsp.add_function(function_id='union', function=lambda *a: ''.join(a),
... inputs=['fname', 'suffix'], outputs=['basename'])
'union'

[graph]

Tip: You can explore the diagram by clicking on it.

Note: For more details how to created a ``Dispatcher`` see:
``add_data()``, ``add_func()``, ``add_function()``,
``add_dispatcher()``, ``SubDispatch``, ``SubDispatchFunction``,
``SubDispatchPipe``, ``DispatchPipe``, and ``DFun``.

The next step to calculate the outputs would be just to run the
``dispatch()`` method. You can invoke it with just the inputs, so it
will calculate all reachable outputs:

..

>>> inputs = {'path': 'schedula/_version.py'}
>>> o = dsp.dispatch(inputs=inputs)
>>> o
Solution([('path', 'schedula/_version.py'),
('basename', '_version.py'),
('dirname', 'schedula'),
('fname', '_version'),
('suffix', '.py')])

[graph]

or you can set also the outputs, so the dispatch will stop when it
will find all outputs:

..

>>> o = dsp.dispatch(inputs=inputs, outputs=['basename'])
>>> o
Solution([('path', 'schedula/_version.py'), ('basename', '_version.py')])

[graph]