titanfe 0.1.3a66

titan Data Flow Engine for Python

Titan Data Flow Engine

The Titan flow engine provides the backend services for running data flows on a node.

The core flow engine is composed of

• Control Peer: One peer per node that takes care of managing Brick Runners on that node.
• Brick Runner: Virtual Machine for executing the Bricks that the Control Peer has started on it's node. The Brick Runner is generic and can process all types of Bricks. Each runner will only execute one type of Brick at a time.

Additionally Version 0.1.3 and onward require the Flow Driector to run.

If you want to run a local self contained version please use the Version 0.1.2x of the Data Flow engine.

Running the Control Peer

Parameters for starting the Control Peer on the command line are:

Parameter	Opt	Description
brick_config		Path to Brick configuration file
kafka <hostname:port>	x	Network address of Kafka instance for metrics and logging
		(default: localhost:9092)
flow_director <hostname:port>	x	Network address of the FlowDirector
		(default: localhost:8080)

To try out the flow engine first download and start the FlowDirector

go build .cmd/flowdirector
./flowdirector.exe

Afterwards download an example Brick configuration file (demo_flow.yml) and store it in an arbitrary folder. Then, open a command line and run the demo flow from within that folder:

python -m titanfe.apps.control_peer -kafka localhost:9092 -flow_director localhost:8080 -brick_config demo_flow.yml

Or run it from the root directory of the flow engine by providing the path to the example:

python -m titanfe.apps.control_peer -kafka localhost:9092 -flow_director localhost:8080 -brick_config path_to_example/demo_flow.yml

Installing the Titan flow engine

The Titan flow engine is available on the Python Package Index (PyPi). It is available for Linux and Windows x64 systems. To install it without having to build it yourself open a command line and run:

pip install titanfe

Building the Titan flow engine

Required packages for building or developing the project can be installed via the requirements.txt in the project's root folder:

pip install -r requirements.txt

To build and install open a command line and run:

python setup.py build
pip install .

Code Quality

Code quality within the project is checked using pylint and flake8.

pylint

Linting is performed with pylint. To define the intended checks .pylintrc is used to configure linting for this project.

Running pylint for the python code in this project the following commands are used:

pylint --rcfile=.pylintrc titanfe

Linting the tests is done running the command:

pylint --rcfile=.pylintrc --disable=duplicate-code ./test

flake8

To make sure the PEP8 standard is applied to the code flake8 can be added to the static tests.

For this project we exclude various errors, warnings and notifications because they do not make sense at this time. This may change while refactoring is considered.

You can run flake 8 with:

flake8

It finds all the python files in this project. The configuration for this project is read from .flake8 in the project's root directory.