titan Data Flow Engine for Python
Project description
Titan Data Flow Engine
The Titan flow engine provides the backend services for running data flows on a node.
The core flow engine is composed of
- Control Peer: One peer per node that takes care of managing Brick Runners on that node.
- Brick Runner: Virtual Machine for executing the Bricks that the Control
Peer has started on it's node. The Brick Runner is generic and can process all types of Bricks.
Each runner will only execute one type of Brick at a time.
Running the Control Peer
Parameters for starting the Control Peer on the command line are:
Parameter | Opt | Description |
---|---|---|
flow_config | Path to Control Peer and Flow configuration file | |
kafka <hostname:port> | x | Network address of Kafka instance for metrics and logging |
(default: localhost:9092) |
To try out the flow engine download an example flow
and store it in an arbitrary folder.
Then, open a command line and run the demo flow from within that folder:
python -m titanfe.apps.control_peer -kafka localhost:9092 -flow_config demo_flow.yml
Or run it from the root directory of the flow engine by providing the path to the example:
python -m titanfe.apps.control_peer -kafka localhost:9092 -flow_config path_to_example/demo_flow.yml
Installing the Titan flow engine
The Titan flow engine is available on the Python Package Index (PyPi). It is available
for Linux and Windows x64 systems. To install it without having to build it yourself
open a command line and run:
pip install titanfe
Building the Titan flow engine
Required packages for building or developing the project can be installed
via the requirements.txt
in the project's root folder:
pip install -r requirements.txt
To build and install open a command line and run:
python setup.py build
pip install .
Code Quality
Code quality within the project is checked using pylint and flake8.
pylint
Linting is performed with pylint. To define the
intended checks .pylintrc
is used to configure linting for this project.
Running pylint for the python code in this project the following commands are
used:
pylint --rcfile=.pylintrc titanfe
Linting the tests is done running the command:
pylint --rcfile=.pylintrc --disable=duplicate-code ./test
flake8
To make sure the PEP8 standard is applied to the code flake8 can be added to
the static tests.
For this project we exclude various errors, warnings and notifications because
they do not make sense at this time. This may change while refactoring is
considered.
You can run flake 8 with:
flake8
It finds all the python files in this project.
The configuration for this project is read from .flake8
in the project's
root directory.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distributions
Built Distribution
Hashes for titanfe-0.1.2a58-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 98c6213680ccfc7999a1e22a58945dabc027870f728533065157494d3d555d2f |
|
MD5 | 73fc808c17c55299ef9a6167b319c7c6 |
|
BLAKE2b-256 | 2b3e78c126551c7c3c27e166a9388771e993d87859fd3cc1d96d215499b8d1af |