Skip to main content

Queue service built on top of mongo.

Project description

mongo_queue

Task queue built on mongo with channels and unique job id.

WebsiteautobotAI Cloud Governance

Inspired from kapilt/mongoqueue

Change Log:

v0.0.3

  • Added unique index with job_id and channel. This is to make sure that the same job is not added multiple times. If not job id provided an unique id generated by default.

Usage

Install the package.

pip install mongo_queue

Usage Example:

  • Create Queue Object
from mongo_queue.queue import Queue
from pymongo import MongoClient

queue = Queue(MongoClient('localhost', 27017).task_queue, consumer_id="consumer-1", timeout=300, max_attempts=3)
  • Add task to queue default channel
queue.put({"task_id": 1})
  • Add task to queue with priority to default channel
queue.put({"task_id": 1}, priority=1)
  • Add task to queue in a specific channel
queue.put({"task_id": 1}, priority=1, channel="channel_1")
  • Add task to queue with unique job_id
queue.put({"task_id": 1}, priority=1, channel="channel_1", job_id="x_job")
  • Get the next job to be executed from the default channel
job = queue.next()
  • Get the next job to be executed from a specific channel
job = queue.next(channel="channel_1")
  • Update job progress for long running jobs
job.progress(count=10)
  • Put the job back in queue, this will be picked up again later, this will update attempts after max attempts the job will not be picked up again.
job.release()
  • Put the job back in queue with error, this will be picked up again later, this will update attempts after max attempts the job will not be picked up again.
job.error("Some error occured")
  • Complete the job. This will delete job from the database.
job.complete()

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for mongo-queue-service, version 0.0.3
Filename, size File type Python version Upload date Hashes
Filename, size mongo_queue_service-0.0.3-py3-none-any.whl (7.5 kB) File type Wheel Python version py3 Upload date Hashes View hashes
Filename, size mongo_queue_service-0.0.3.tar.gz (5.3 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page