Skip to main content

A message queue system using AWS SQS

Project description

SQS-MQ

A message queue system using AWS SQS

Features

  • Automatically process messages
  • Created completed and failed buckets
  • Create bucket by pool name
  • Process messages by bucket

Install

pip install sqs-mq

Create/Connect

mq = SQSMQ(name, 
          aws_access_key_id, 
          aws_secret_access_key, 
          region, 
          visibility_timeout)

Alternatively, connect later

mq = SQSMQ(name)

mq.connect(aws_access_key_id, 
          aws_secret_access_key, 
          region)

SQS-MQ will create 3 buckets of messages:

QUEUED: This bucket will contain all the messages to be processed

COMPLETED: contains all messages that have have processed successfully

FAILED: contains all messages that have failed to process properly, or due to an error.

mq = SQSMQ("myQname", **kwargs)

Three buckets will be created: myQname-QUEUED, myQname-COMPLETED, myQname-FAILED


Usage

Add Task

mq.add(my_function_name, *args, **kwargs)

Add Task in different pool

To add a task in a different pool. it will be named: myQname-QUEUED-FAST

mq.add(my_fn, *args, **kwargs, pool='FAST')

Add delay

To add a delay

mq.add(my_fn, delay=45)

Run the tasks

Run all

mq.run()

Run by pool name

mq.run(pool='FAST')

COMPLETED and FAILED

Once a tasks have been processed, it will be added in either COMPLETED or FAILED buckets.

Run all items in the COMPLETED bucket

def completed_callback(body:dict): 
  data = ...

mq.run_completed(callback=completed_callback)

Run all items in the FAILED bucket

def failed_callback(body:dict): 
  data = ...

mq.run_failed(callback=failed_callback)

Once captured, it will delete the message from the queue

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

sqs-mq-0.2.1.tar.gz (4.5 kB view hashes)

Uploaded source

Built Distribution

sqs_mq-0.2.1-py2-none-any.whl (4.3 kB view hashes)

Uploaded py2

Supported by

AWS AWS Cloud computing Datadog Datadog Monitoring Facebook / Instagram Facebook / Instagram PSF Sponsor Fastly Fastly CDN Google Google Object Storage and Download Analytics Huawei Huawei PSF Sponsor Microsoft Microsoft PSF Sponsor NVIDIA NVIDIA PSF Sponsor Pingdom Pingdom Monitoring Salesforce Salesforce PSF Sponsor Sentry Sentry Error logging StatusPage StatusPage Status page