Skip to main content
Join the official 2019 Python Developers SurveyStart the survey!

3 stage multithread engine.

Project description

# os-m3-engine

[![Build Status](https://www.travis-ci.org/cfhamlet/os-m3-engine.svg?branch=master)](https://www.travis-ci.org/cfhamlet/os-m3-engine)
[![codecov](https://codecov.io/gh/cfhamlet/os-m3-engine/branch/master/graph/badge.svg)](https://codecov.io/gh/cfhamlet/os-m3-engine)
[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/os-m3-engine.svg)](https://pypi.python.org/pypi/os-m3-engine)
[![PyPI](https://img.shields.io/pypi/v/os-m3-engine.svg)](https://pypi.python.org/pypi/os-m3-engine)

Multi-thread engine for 3(or 2) stages job.



Although multi-thread of python is buggy and slow, it is still necessary to write multi-thread program. Typically, producer-consumer is the most common model(so called 2-stage job), further more a transporter is needed between them(3-stage job). This library is used to simplify creating 3(or 2) stages program.



The 3 stages are:

* Frontend: think as producer, usually used to create or receive data.
* Transport: receive data from frontend stage, transform and send the transformed data to backend.
* Backend: think as consumer, process the data received from transport.



Something else need to know:

* Each of the stage can be multi-thread.
* Frontend is required, transport or backend can be omitted.




# Install

`pip install os-m3-engine`

# API

* Create default engine, a typical 3-stage job:

- frontend: ``os_m3_engine.ootb.StdinFrontend``, read from stdin, send to transport stage
- transport: ``os_m3_engine.ootb.LogTransport``, log data received from fronted, send to backend stage
- backend: ``os_m3_engine.ootb.LogBackend``, log data received from transport

```
from os_m3_engine.launcher import create

engine = create()
```

* Create engine with custom defined stage:

```
from os_m3_engine.launcher import create

engine = create(transport_cls='transport_class_path_or_class')
```

* Create engine with custom engine config:

```
from os_m3_engine.launcher import create

config = WhateverOjbectYouWant
config.thread_num = 10
engine = create(transport_cls='your_transport_cls', engine_transport_config=config)
```

* Start the engine:

``start`` will block the current thread until all of the stage threads stopped

```
engine.start()
```

* The regular practice of stopping the engine

```
from signal
from os_m3_engine.launcher import create

engine = create()

def stop(signum, frame):
engine.stop()

signal.signal(signal.SIGINT, stop)

engine.start()
```

* Custom frontend class:

- inherit from ``os_m3_engine.core.frontend.Frontend``
- define ``produce`` method as generator

```
from os_m3_engine.core.frontend import Frontend

class CustomFrontend(Frontend):
def produce(self):
yield 'Hello world!'
```

* Custom transport class:

- inherit from ``os_m3_engine.core.transport.Transport``
- define ``transport`` method, the only parameter is the data received, the return value will be sent to backend

```
from os_m3_engine.core.transport import Transport

class CustomTransport(Transport):
def transport(self, data):
return data
```

* Custom backend class:

- inherit from ``os_m3_engine.core.backend.Backend``
- define ``process`` method, the only parameter is the data received, no need return

```
from os_m3_engine.core.backend import Backend

class CustomBackend(Backend):
def process(self, data):
print(data)
```

* Passing parameters

- create engine with custom config object
- use ``self.config`` to get the config in stage class

```
from os_m3_engine.launcher import create

config = WhateverOjbectYouWant
engine = create(app_config=config)
```


* Custom ``setup``, ``cleanup`` behavior

- each stage class can define ``setup``, ``cleanup`` methods
- these will be called at each thread start/stop


```
from os_m3_engine.core.backend import Backend

class CustomBackend(Backend):

def setup(self):
print('Setup')

def cleanup(self):
print('Cleanup')

def process(self, data):
print(data)
```



# Unit Tests

`$ tox`

# License

MIT licensed.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for os-m3-engine, version 0.1.2
Filename, size File type Python version Upload date Hashes
Filename, size os-m3-engine-0.1.2.tar.gz (10.4 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page