Azkaban Orchestrator
Project description
Azkaban Orchestrator
====================
An Orchestrator for Azkaban pipelines
Quick Start
----------
.. code-block:: console
pip install azkaban-orchestrator
Define Orchestration logic
--------------------------
Create a file and define the dependencies between pipelines using the following notation.
* Pipeline name is a project name in Azkaban. Project and flow name of a pipeline should be the same.
* use '->' for **hard dependency** between two pipelines. A -> B means B runs if A runs successfully.
* use '.>' for **soft dependency** between two pipelines. A .> B means B runs after A despite A runs successfully or not.
* if pipeline has some **parameters** put them in parentheses by the pipeline name like A(status=1|date). use vertical bar '|' to separate the parameters.
* to define **clusters** use colon ':' e.g. S: A,B,C means cluster S includes pipelines A,B and C. to move from one cluster all the pipelines within the cluster should run successfully.
Sample 1
--------
.. code-block:: console
----------
| a b | s1
----------
|
v
----------
| c d | s2
----------
|
v
e
Sample1 diagram file
.. code-block:: console
s1 : a, b
s2 : c, d
s1 -> s2
s2 -> e
Sample 2
--------
.. code-block:: console
a -> b -> d(date)
| ___/|
| | |
v v v
c -> e <- f(status=1)
|
v
f(status=2)
Sample2 diagram file
.. code-block:: console
a -> b
b -> d(date)
d(date) -> f(status=1)
d(date) -> e
f(status=1) -> e
a -> c
c -> e
c -> f(status=2)
Usage
-----
To run the orchestrator
.. code:: python
import logging
from azkaban_orchestrator import orchestrator
client = orchestrator.Client(
diagram_file_name='/path/to/diargam_file',
host='azkaban_host',
username='azkaban_username',
password='azkaban_passwpord',
logger=logging.getLogger(__name__)
)
# define the parameters need to pass to the orchestrator
params = {'date':'20171202'}
# define the initial pipeline
# if you need to start orchestration from a specific pipeline
initial = None
client.run(initial, params)
To draw the diagram
.. code:: python
from azkaban_orchestrator import diagram
d = diagram.Diagram('test diagram', 'path/to/diagram_file')
d.show()
====================
An Orchestrator for Azkaban pipelines
Quick Start
----------
.. code-block:: console
pip install azkaban-orchestrator
Define Orchestration logic
--------------------------
Create a file and define the dependencies between pipelines using the following notation.
* Pipeline name is a project name in Azkaban. Project and flow name of a pipeline should be the same.
* use '->' for **hard dependency** between two pipelines. A -> B means B runs if A runs successfully.
* use '.>' for **soft dependency** between two pipelines. A .> B means B runs after A despite A runs successfully or not.
* if pipeline has some **parameters** put them in parentheses by the pipeline name like A(status=1|date). use vertical bar '|' to separate the parameters.
* to define **clusters** use colon ':' e.g. S: A,B,C means cluster S includes pipelines A,B and C. to move from one cluster all the pipelines within the cluster should run successfully.
Sample 1
--------
.. code-block:: console
----------
| a b | s1
----------
|
v
----------
| c d | s2
----------
|
v
e
Sample1 diagram file
.. code-block:: console
s1 : a, b
s2 : c, d
s1 -> s2
s2 -> e
Sample 2
--------
.. code-block:: console
a -> b -> d(date)
| ___/|
| | |
v v v
c -> e <- f(status=1)
|
v
f(status=2)
Sample2 diagram file
.. code-block:: console
a -> b
b -> d(date)
d(date) -> f(status=1)
d(date) -> e
f(status=1) -> e
a -> c
c -> e
c -> f(status=2)
Usage
-----
To run the orchestrator
.. code:: python
import logging
from azkaban_orchestrator import orchestrator
client = orchestrator.Client(
diagram_file_name='/path/to/diargam_file',
host='azkaban_host',
username='azkaban_username',
password='azkaban_passwpord',
logger=logging.getLogger(__name__)
)
# define the parameters need to pass to the orchestrator
params = {'date':'20171202'}
# define the initial pipeline
# if you need to start orchestration from a specific pipeline
initial = None
client.run(initial, params)
To draw the diagram
.. code:: python
from azkaban_orchestrator import diagram
d = diagram.Diagram('test diagram', 'path/to/diagram_file')
d.show()
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Close
Hashes for azkaban-orchestrator-0.0.1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9bab558b46cf2a3dcf58e3a1b079f8995deb2a5928069a0115adbb785ed518df |
|
MD5 | 8494a64f25ad997cbc0126fdc4881327 |
|
BLAKE2b-256 | 7f5b8a3da6d982c08892509e86be4daafea976b4344ac7095a94232c03ae5d15 |