Skip to main content

3 stage multithread engine.

Project description

# os-m3-engine

[![Build Status](https://www.travis-ci.org/cfhamlet/os-m3-engine.svg?branch=master)](https://www.travis-ci.org/cfhamlet/os-m3-engine)
[![codecov](https://codecov.io/gh/cfhamlet/os-m3-engine/branch/master/graph/badge.svg)](https://codecov.io/gh/cfhamlet/os-m3-engine)
[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/os-m3-engine.svg)](https://pypi.python.org/pypi/os-m3-engine)
[![PyPI](https://img.shields.io/pypi/v/os-m3-engine.svg)](https://pypi.python.org/pypi/os-m3-engine)

Multi-thread engine for 3(or 2) stages job.



Although multi-thread of python is buggy and slow, it is still necessary to write multi-thread program. Typically, producer-consumer is the most common model(so called 2-stage job), further more a transporter is needed between them(3-stage job). This library is used to simplify creating 3(or 2) stages program.



The 3 stages are:

* Frontend: think as producer, usually used to create or receive data.
* Transport: receive data from frontend stage, transform and send the transformed data to backend.
* Backend: think as consumer, process the data received from transport.



Something else need to know:

* Each of the stage can be multi-thread.
* Frontend is required, transport or backend can be omitted.




# Install

`pip install os-m3-engine`

# API

* Create default engine, a typical 3-stage job:

- frontend: ``os_m3_engine.ootb.StdinFrontend``, read from stdin, send to transport stage
- transport: ``os_m3_engine.ootb.LogTransport``, log data received from fronted, send to backend stage
- backend: ``os_m3_engine.ootb.LogBackend``, log data received from transport

```
from os_m3_engine.launcher import create

engine = create()
```

* Create engine with custom defined stage:

```
from os_m3_engine.launcher import create

engine = create(transport_cls='transport_class_path_or_class')
```

* Create engine with custom engine config:

```
from os_m3_engine.launcher import create

config = WhateverOjbectYouWant
config.thread_num = 10
engine = create(transport_cls='your_transport_cls', engine_transport_config=config)
```

* Start the engine:

``start`` will block the current thread until all of the stage threads stopped

```
engine.start()
```

* The regular practice of stopping the engine

```
from signal
from os_m3_engine.launcher import create

engine = create()

def stop(signum, frame):
engine.stop()

signal.signal(signal.SIGINT, stop)

engine.start()
```

* Custom frontend class:

- inherit from ``os_m3_engine.core.frontend.Frontend``
- define ``produce`` method as generator

```
from os_m3_engine.core.frontend import Frontend

class CustomFrontend(Frontend):
def produce(self):
yield 'Hello world!'
```

* Custom transport class:

- inherit from ``os_m3_engine.core.transport.Transport``
- define ``transport`` method, the only parameter is the data received, the return value will be sent to backend

```
from os_m3_engine.core.transport import Transport

class CustomTransport(Transport):
def transport(self, data):
return data
```

* Custom backend class:

- inherit from ``os_m3_engine.core.backend.Backend``
- define ``process`` method, the only parameter is the data received, no need return

```
from os_m3_engine.core.backend import Backend

class CustomBackend(Backend):
def process(self, data):
print(data)
```

* Passing parameters

- create engine with custom config object
- use ``self.config`` to get the config in stage class

```
from os_m3_engine.launcher import create

config = WhateverOjbectYouWant
engine = create(app_config=config)
```


* Custom ``setup``, ``cleanup`` behavior

- each stage class can define ``setup``, ``cleanup`` methods
- these will be called at each thread start/stop


```
from os_m3_engine.core.backend import Backend

class CustomBackend(Backend):

def setup(self):
print('Setup')

def cleanup(self):
print('Cleanup')

def process(self, data):
print(data)
```



# Unit Tests

`$ tox`

# License

MIT licensed.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

os-m3-engine-0.1.2.tar.gz (10.4 kB view hashes)

Uploaded Source

Supported by

AWS AWS Cloud computing and Security Sponsor Datadog Datadog Monitoring Fastly Fastly CDN Google Google Download Analytics Microsoft Microsoft PSF Sponsor Pingdom Pingdom Monitoring Sentry Sentry Error logging StatusPage StatusPage Status page