Flask-Multiprocess-Controller 0.1.1

Flask extension that provides an easy-to-implement controller for multiprocessing tasking. It provides default functions such as task-queueing, health-check, status-check, manual-stop and process-safe logger.

Flask-Multiprocess-Controller is an extension for Flask that provides an easy-to-implement controller for multiprocessing tasking. It provides default functions such as task-queueing, health-check, status-check, manual-stop and process-safe logger.

A common usage scenario is one has some tasks which are computational expensive and require multiple instances running concurrently. This controller provides a separate layer of controller by creating controlling thread for each instance. This can greatly improve level-of-control for computational expensive tasks.

Features

Flask-Multiprocess-Controller provides the MetaMPController() class which provides default controlling method to control the execute() method in the linked, child of MetaMPTask() class. Default controlling methods are as follows: - GET: retrieve the current status of certain task in sub-process - POST: put an execution request to the queue waiting to be executed when there is a new slot for a new sub-process - DELETE: sending the stop signal to certain task in sub-process gently

All Flask supported HTTP methods are supported(with necessary overrides)

One can also use MetaMPResource() class to adopt Flask-RESTful style APIs.

Example: Build a minimal Controller-Task structure

In the following example, we will use the built-in TemplateMPController, and TemplateMPResource to build the structure of controller-task framework. One may need to override some methods in MetaMPController for more functionality.

import logging
from flask import Flask
from flask_restful import Api
from flask_multiprocess_controller import TemplateFactory, MetaMPTask

class SampleTask(MetaMPTask):

    def execute(self) -> None:
        task_logger = logging.getLogger(str(os.getpid()))
        counter = 0
        while counter < 100:
            counter += 1
            task_logger.debug("doing some simple task, progress {}%.".format(counter))
            time.sleep(0.5)
            self.upload_status(counter)
            self.checkpoint()

sample_api = Api()
sample_controller = TemplateFactory.MPController(name='Sample')(target_task=SampleTask, max_num_process=2)
sample_api.add_resource(TemplateFactory.MPResource(), '/sample', resource_class_args=(sample_controller,))

app = Flask('Sample')
sample_api.init_app(app)
app.run()

Installation

The easiest way to install it is using pip from PyPI

pip install flask-multiprocess-controller

License

See the LICENSE file for license rights and limitations (BSD-3-Clause).