Logging handlers to AWS services that support S3 and Kinesis stream logging with multiple threads
Project description
A python multithreaded logging handler package that streams records to AWS services objects with support for the following AWS services: * S3 * Kinesis
Getting Started
Prerequisites
Asynchronous multipart uploading relies on the ability to use multiple threads #### Packages:
boto3
Installing
Installation using pip
pip install aws-logging-handlers
Examples
Stream log records to S3 and Kinesis
import logging from logging_s3_handler import S3Handler KEY_ID="your_aws_auth_key" SECRET="your_aws_auth_secret" bucket="test_bucket" # The bucket should already exist # The log will be rotated to a new object either when an object reaches 5 MB or when 120 seconds pass from the last rotation/initial logging s3_handler = S3Handler("test_log", bucket, KEY_ID, SECRET, time_rotation=120, max_file_size_bytes=5*1024**2, workers=3) kinesis_handler = KinesisHandler(KEY_ID, SECRET, 'log_test', 'us-east-1', partition='test1', workers=1) formatter = logging.Formatter('[%(asctime)s] {%(filename)s:%(lineno)d} %(levelname)s - %(message)s') s3_handler.setFormatter(formatter) kinesis_handler.setFormatter(formatter) logger = logging.getLogger('root') logger.setLevel(logging.INFO) logger.addHandler(s3_handler) logger.addHandler(kinesis_handler) for i in range(0, 100000): logger.info("test info message") logger.warning("test warning message") logger.error("test error message")
To be developed
Support for asyncio
Logging and upload metrics
support for real time compression(at least for s3)
License
This project is licensed under the MIT License - see the LICENSE.md file for details
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Close
Hashes for aws-logging-handlers-0.1.4.1.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 21b47cec615cbc6b3dc73a9bf7c07d35b8bffaf9dba6450106a4790b624e4649 |
|
MD5 | 417abe3496bbc5a93e4c7a1d2a452c77 |
|
BLAKE2b-256 | c195ab9ce2816cb3c07cb1ef312ec51cf0008735bcd5e3c1ac6a0b744d893520 |