sidita 1.0.1

Simple Distributed Task Queue : multiprocessing < sidita < celery

• Quick Link to Production Branch

• Quick Link to Devel Branch

Overview

What is sidita ?

Sidita is a Python module which implements a distributed task queue featuring an intermediate solution between the multiprocessing module and a task scheduler like Celery.

The Sidita use case corresponds to the case where you need to run CPU bound tasks in parallel and you require an immunity to crashes, memory leaks and overruns. Theses requirements are met by a monitored subprocess implementation.

Where is the Documentation ?

The documentation is available on the sidita Home Page.

What are the main features ?

• Must be a simple solution to circumvent to the GIL limitation : i.e. distribute and run tasks in parallel on local CPU cores

• Must be a lightweight solution to Celery : no broker

• Should be portable on main OS like Unix and Windows

• Scheduler is implemented using an asyncio event loop and asyncio queue

• Scheduler queue can be limited on size so as to prevent a memory overshoot at startup

• Workers are spawned using asyncio subprocess and communicate through a pipe (stdin, stdout)

• Pass any pickable object as request and response

• Scheduler monitors workers and restart them when they was killed or they exceeded a memory or timeout threshold

• Provide basic metrics on workers

Technical details in brief:

• scheduler run a producer and N consumers coroutines in an asyncio event loop

• the producer coroutine awaits on queue.put(task) ( if a size limit is set )

• each consumer coroutine wraps a worker subprocess and await data on the stdout pipe

Technical details in brief:

• scheduler run a producer and N consumers coroutines in an asyncio event loop

• the producer coroutine awaits on queue.put(task) ( if a size limit is set )

• each consumer coroutine wraps a worker subprocess and await data on the stdout pipe

How to install it ?

Look at the installation section in the documentation.

Credits

Authors: Fabrice Salvaire

News

V0 2018-05-01

Started project