titan Data Flow Engine for Python
Titan Flow Engine
The Titan Flow Engine provides the backend services for running data flows on a node.
The core flow engine is composed of two services:
- Control Peer: One peer per node that takes care of managing Brick Runners on that node.
- Brick Runner: Virtual Machine for executing the Bricks that the Control Peer has started on it's node. The Brick Runner is generic and can process all types of Bricks. Each runner will only execute one type of Brick at a time.
Choose How to Install
Titan Flow Engine requires Python 3.7 to be installed.
Depending on whether you want to contribute to the Titan Flow Engine source code you want to choose one of the following options to install:
Installing via PyPi
The flow engine is available on the Python Package Index (PyPi). It is available for Linux and Windows x64 systems. To install it without having to build it yourself open a command line and run:
pip install titanfe
Building the Titan Flow Engine (Advanced Install)
Required packages for building or developing the project can be installed
requirements_dev.txt in the project's root folder:
pip install -r requirements_dev.txt
To build and install the Flow Engine open a command line and run:
python setup.py build pip install .
For those who also want to work on the documentation and build it locally,
please use the
requirements_doc.txt file to install the necessary python
pip install -r requirements_doc.txt
Running the flow engine
The flow engine is being started by starting the Control Peer. The Control Peer takes care of starting Brick Runners as needed.
Parameters for starting the Control Peer on the command line are:
|brick_folder||Folder to store installed bricks in (default: ~/titanfe/bricks)|
|config_file||Path to ControlPeer configuration file|
The ControlPeers configuration file contains the following parameters:
|Kafka||string||Network address of Kafka instance for metrics and logging|
|GridManager||string||Network address of the GridManager|
|FlowManager||string||Network address of the FlowManager|
|EndpointProvider||string||Network address of the EndpointProvider|
|SecretKey||string||secret key for brick parameter decryption|
|(default: None), alternatively use TITAN_SECRET_KEY environment variable|
If no secret key is given, the ControlPeer will stop immediately. The secret key needs to be identical to the one used in the FlowManager for encrypting parameters.
To try out the flow engine first download and start the services GridManager, Repository service, User service, PackageManager, and FlowManager. These services are needed for the flow engine to receive flows to process.
Adjust the ControlPeers configuration file (titanfe/apps/control_peer/config.yaml) or create a new one. Then, open a command line and run.
python -m titanfe.apps.control_peer -brick_folder ~/titanfe/bricks -config_file path_to_config_file/config.yaml
Or run it from the root directory of the flow engine by providing the path to the example:
python -m titanfe.apps.control_peer -brick_folder ~/titanfe/bricks -config_file titanfe/apps/control_peer/config.yaml
Code quality within the project is checked using pylint and flake8.
Linting is performed with pylint. To define the
.pylintrc is used to configure linting for this project.
Running pylint for the python code in this project the following commands are used:
pylint --rcfile=.pylintrc titanfe
Linting the tests is done running the command:
pylint --rcfile=.pylintrc --disable=duplicate-code ./test
To make sure the PEP8 standard is applied to the code flake8 can be added to the static tests.
For this project we exclude various errors, warnings and notifications because they do not make sense at this time. This may change while refactoring is considered.
You can run flake 8 with:
It finds all the python files in this project.
The configuration for this project is read from
.flake8 in the project's
Release history Release notifications | RSS feed
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Hashes for titanfe-0.4.179-py3-none-any.whl