Skip to main content

Simple Distributed Task Queue : multiprocessing < sidita < celery

Project description

sidita license sidita python version

sidita last version

Overview

What is sidita ?

Sidita is a Python module which implements a distributed task queue featuring an intermediate solution between the multiprocessing module and a task scheduler like Celery.

The Sidita use case corresponds to the case where you need to run CPU bound tasks in parallel and you require an immunity to crashes, memory leaks and overruns. Theses requirements are met by a monitored subprocess implementation.

Where is the Documentation ?

The documentation is available on the sidita Home Page.

What are the main features ?

  • Must be a simple solution to circumvent to the GIL limitation : i.e. distribute and run tasks in parallel on local CPU cores
  • Must be a lightweight solution to Celery : no broker
  • Should be portable on main OS like Unix and Windows
  • Scheduler is implemented using an asyncio event loop and asyncio queue
  • Scheduler queue can be limited on size so as to prevent a memory overshoot at startup
  • Workers are spawned using asyncio subprocess and communicate through a pipe (stdin, stdout)
  • Pass any pickable object as request and response
  • Scheduler monitors workers and restart them when they was killed or they exceeded a memory or timeout threshold
  • Provide basic metrics on workers

Technical details in brief:

  • scheduler run a producer and N consumers coroutines in an asyncio event loop
  • the producer coroutine awaits on queue.put(task) ( if a size limit is set )
  • each consumer coroutine wraps a worker subprocess and await data on the stdout pipe

Technical details in brief:

  • scheduler run a producer and N consumers coroutines in an asyncio event loop
  • the producer coroutine awaits on queue.put(task) ( if a size limit is set )
  • each consumer coroutine wraps a worker subprocess and await data on the stdout pipe

How to install it ?

Look at the installation section in the documentation.

Credits

Authors: Fabrice Salvaire

News

V0 2018-05-01

Started project

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for sidita, version 1.0.1
Filename, size File type Python version Upload date Hashes
Filename, size sidita-1.0.1-py2.py3-none-any.whl (15.9 kB) File type Wheel Python version py2.py3 Upload date Hashes View

Supported by

Pingdom Pingdom Monitoring Google Google Object Storage and Download Analytics Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page