The qing of pipes
Project description
Framework for building Data pipelines that scales.
Installation
====
```
pip install qing-framework
```
Usage
=====
Build your first worker
-----------------------
``` python
# normalizer_worker.py
from qing_framework import QingWorker
from qing_framework import QingMessage as QM
class Normalizer(QingWorker):
def process(self, qs):
#do your magic
formatted_messages = []
# messages from queue named 'dummy' are loaded for you automagically inside the qs variable.
while message in qs['dummy']:
# Each QingMessage has a payload and now you can create your own after modifying!
formatted_message = QM( payload="formatted: " + message.payload )
formatted_messages.append(formatted_message)
#returned messages are automatically writted to output queue!
return formatted_messages
```
``` python
# normalizer_manifest.py
# This file describes worker's responsibility and dependencies.
{
'name':'normalizer',
'author': 'AlSayed Gamal',
'version': '0.0.1',
'description': 'normalizes messages into proper format!',
'in-queues': ['dummy'],
'out-queues': ['normalized'],
'activity-rate': '5000',
'class': 'Normalizer'
}
```
Run worker using CLI
--------------------
You can start qing as a server and use qing's CLI to run and scale the worker
```bash
#queen cli and UI for qing framework.
qing --help # just in case you needed help.
qingd # qing deamon, you will need sudo permission
qing run Normalizer # run a specific worker
qing kill Normalizer --job-id=123 #kill specific worker job
```
Monitor the data-pipeline using Web-UI
--------------------------------------
```bash
qing --webserver --port=8910 # you can view data pipelines, queues, jobs and workers.
```
Installation
====
```
pip install qing-framework
```
Usage
=====
Build your first worker
-----------------------
``` python
# normalizer_worker.py
from qing_framework import QingWorker
from qing_framework import QingMessage as QM
class Normalizer(QingWorker):
def process(self, qs):
#do your magic
formatted_messages = []
# messages from queue named 'dummy' are loaded for you automagically inside the qs variable.
while message in qs['dummy']:
# Each QingMessage has a payload and now you can create your own after modifying!
formatted_message = QM( payload="formatted: " + message.payload )
formatted_messages.append(formatted_message)
#returned messages are automatically writted to output queue!
return formatted_messages
```
``` python
# normalizer_manifest.py
# This file describes worker's responsibility and dependencies.
{
'name':'normalizer',
'author': 'AlSayed Gamal',
'version': '0.0.1',
'description': 'normalizes messages into proper format!',
'in-queues': ['dummy'],
'out-queues': ['normalized'],
'activity-rate': '5000',
'class': 'Normalizer'
}
```
Run worker using CLI
--------------------
You can start qing as a server and use qing's CLI to run and scale the worker
```bash
#queen cli and UI for qing framework.
qing --help # just in case you needed help.
qingd # qing deamon, you will need sudo permission
qing run Normalizer # run a specific worker
qing kill Normalizer --job-id=123 #kill specific worker job
```
Monitor the data-pipeline using Web-UI
--------------------------------------
```bash
qing --webserver --port=8910 # you can view data pipelines, queues, jobs and workers.
```
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Built Distribution
Close
Hashes for qing_framework-0.0.1rc0-py2-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0b1d79e51cb4d4f0d0eff21ce2eaef8a72803fdc68b9e94e57651b86b1391b27 |
|
MD5 | b1406ce2b8bb7cd60e450f78b8abdf09 |
|
BLAKE2-256 | 4437bada04b2e5429b7ae5e2f4574bf2e6419eaa93812d6477c5faa1ed7f91db |