Skip to main content

Automated serverless logging to S3 via SQS.

Project description

sqs-s3-logger

A library to persist messages on S3 using serverless architecture. It is mainly targeted at cheaply archiving low-volume, sporadic events from applications without a need to spin additional infrastructure.

Overall idea

What it’s not

Not a replacement for general logging systems or libraries. Provides no filtering or aggregation.

AWS Alternatives

Usage

Configure boto3’s credentials as per: http://boto3.readthedocs.io/en/latest/guide/quickstart.html#configuration

Make sure you setup:

  • AWS_ACCESS_KEY_ID
  • AWS_SECRET_ACCESS_KEY
  • AWS_DEFAULT_REGION (optionally)

Take a look at main.py.

For help: python3 main.py -h

For example (backup at midnight each Saturday from app-logs queue to app-logs-archive bucket):

sqs-s3-logger create -b app-logs-archive -q app-logs -f app-logs-backup -s 'cron(0 0 ? * SAT *)'

Sending messages to a queue

Ideally you should use another AWS IAM user with permissions restricted to getting SQS queues and writing messages.

import boto3
sqs = boto3.resource('sqs')
queue = sqs.get_queue_by_name(QueueName='<QUEUE_NAME>')
queue.send_message(MessageBody='<MSG_BODY_NOT_EXCEEDING_256KB>')

Limitations

  • Maximum SQS message size is limited to 256 KB
  • There could be no more than 120,000 messages in a queue at a time.
  • SQS messages cannot persist for longer than 14 days.
  • Lambda environment has up to 512MB of ephemeral disk capacity.
  • By default it does not guarantee correct time-based ordering

You may need to adjust your CRON settings depending on your volume.

Testing

python3 setup.py test

These will use your AWS account to instantiate a temporary integration environment.

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Files for sqs-s3-logger, version 1.0.7
Filename, size File type Python version Upload date Hashes
Filename, size sqs-s3-logger-1.0.7.tar.gz (11.6 kB) File type Source Python version None Upload date Hashes View hashes

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page