Library working with Amazon SQS
Project description
aws-sqs-batchlib for Python
Consume and process Amazon SQS queues in large batches.
Features
- Customizable batch size and batch window to consume and process messages in larger (> 10 message) batches. Collect up-to 10,000 messages from a queue and process them in one go.
Installation
Install from PyPI with pip
pip install aws-sqs-batchlib
or with the package manager of choice.
Usage
import aws_sqs_batchlib
# Consume up-to 100 messages from the given queue, polling the queue for
# up-to 1 second to fill the batch.
res = aws_sqs_batchlib.consume(
queue_url = "https://sqs.eu-north-1.amazonaws.com/123456789012/MyQueue",
batch_size=100,
maximum_batching_window_in_seconds=1,
VisibilityTimeout=300,
)
# Returns messages in the same format as boto3 / botocore SQS Client
# receive_message() method.
assert res == {
'Messages': [
{'MessageId': '[.]', 'ReceiptHandle': 'AQ[.]JA==', 'MD5OfBody': '[.]', 'Body': '[.]'},
{'MessageId': '[.]', 'ReceiptHandle': 'AQ[.]wA==', 'MD5OfBody': '[.]', 'Body': '[.]'}
# ... up-to 1000 messages
]
}
Development
Requires Python 3 and Poetry. Useful commands:
# Run tests
poetry run tox -e test
# Run linters
poetry run tox -e lint
# Format code
poetry run tox -e format
Benchmarks & Manual Testing
Use benchmark/benchmark.py
to benchmark and test the library functionality and performance. Execute following commands in Poetry virtualenv (execute poetry shell
to get there):
# Setup
export PYTHONPATH=$(pwd)
export AWS_DEFAULT_REGION=eu-north-1
# Send messages to a queue
python3 benchmark/benchmark.py \
--queue-url https://sqs.eu-north-1.amazonaws.com/123456789012/MyQueue producer
# Consume messages with the plain SQS ReceiveMessage polling
python3 benchmark/benchmark.py \
--queue-url https://sqs.eu-north-1.amazonaws.com/123456789012/MyQueue consumer-plain
# Consume messages with the libary
python3 benchmark/benchmark.py \
--queue-url https://sqs.eu-north-1.amazonaws.com/123456789012/MyQueue consumer-lib \
--batch-size 1000
--batch-window 1
Single thread is able to receive / send around 400 messages per second to an SQS queue on the same AWS region (eu-north-1, m5.large instance).
License
MIT.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
aws-sqs-batchlib-0.1.1.tar.gz
(3.2 kB
view hashes)
Built Distribution
Close
Hashes for aws_sqs_batchlib-0.1.1-py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0055d62c650e4889fd11c05714dd3380f1c1a5b4f17dfde4149daf536e71746c |
|
MD5 | 59390431df902f8a58ca83b347b8e706 |
|
BLAKE2b-256 | b7eedb32562d63c498b44df8e2d38517870f36062b4013b55edd0aa232dc0269 |