Push logs to S3
Project description
Get files from directory by mask and push them to S3
Install
pip install s3logs
Usage
s3logs config.conf
Where config.conf can use that structure:
[S3]
access_key = <S3_KEY>
secret_key = <S3_SECRET_KEY>
host = <s3.example.com>
bucket = <bucket_name>
chunk_size = <bytes, default=52428800>
[logs]
suffix = .0.gz
key_suffix = .gz
directory = /var/log/nginx/
filename = <filename, default=(yesterday as yyyy-mm-dd)>
[map]
example.com-access.log = example/access
example.com-error.log = example/error
mysite.me.access.log = mysite/access
When it used with that config, script takes all files in directory /var/log/nginx/, filter only those, which ends with .0.gz and send it to S3, according to map.
For example, /var/log/nginx now consists of:
example.com-access.log
example.com-access.log.0.gz
example.com-access.log.1.gz
example.com-error.log
example.com-error.log.0.gz
example.com-error.log.1.gz
mysite.me.access.log
mysite.me.access.log.0.gz
mysite.me.error.log
mysite.me.error.log.0.gz
So, if today is 9 December 2015, and your hostname is node1, on your S3 <bucket_name> would be those keys:
node1/example/access/2015-12-09.gz
node1/example/error/2015-12-09.gz
node1/mysite/access/2015-12-09.gz
Because we have not explain how maps mysite.me.error.log.0.gz - it would be skipped.
If there was filename=newfile option in config.conf, keys in S3 would look like node1/example/access/newfile.gz
Project details
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
s3logs-1.4.tar.gz
(3.3 kB
view hashes)
Built Distribution
Close
Hashes for s3logs-1.4.macosx-10.11-x86_64.tar.gz
Algorithm | Hash digest | |
---|---|---|
SHA256 | 5cd691260978599d9bb8c07c83ff8168ee5ab80acd674a435dbb1b625a6a901d |
|
MD5 | a3aa24a790a6739de7ec0443a841e358 |
|
BLAKE2b-256 | e8782fa0ec41a8128b21fc4a334c4ce4bd2540f4d4ddda0dfdf72103f60cbcc5 |