Cloud Custodian - Cloud Watch Log S3 exporter
c7n-log-exporter: Cloud watch log exporter automation
A small serverless app to archive cloud logs across accounts to an archive bucket. It utilizes cloud log export to s3 feature for historical exports.
It also supports kinesis streams / firehose to move to realtime exports in the same format as the periodic historical exports.
- Log group filtering by regex
- Incremental support based on previously synced dates
- Incremental support based on last log group write time
- Cross account via sts role assume
- Lambda and CLI support.
- Day based log segmentation (output keys look like $prefix/$account_id/$group/$year/$month/$day/$export_task_uuid/$stream/$log)
- The archive bucket has already has appropriate bucket policy permissions. For details see: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/S3ExportTasks.html#S3Permissions
- Default periodicity for log group archival into s3 is daily.
- Exporter is run with account credentials that have access to the archive s3 bucket.
- Catch up archiving is not run in lambda (do a cli run first)
You can run on a single account / log group via the export subcommand
c7n-log-exporter export --help
To ease usage when running across multiple accounts, a config file can be specified, as an example.
destination: bucket: custodian-log-archive prefix: logs2 accounts: - name: custodian-demo role: "arn:aws:iam::111111111111:role/CloudCustodianRole" groups: - "/aws/lambda/*" - "vpc-flow-logs"
Multiple accounts via cli
To run on the cli across multiple accounts, edit the config.yml to specify multiple accounts and log groups.
c7n-log-exporter run --config config.yml
Edit config.yml to specify the accounts, archive bucket, and log groups you want to use.
make install make deploy
Release history Release notifications | RSS feed
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
|Filename, size||File type||Python version||Upload date||Hashes|
|Filename, size c7n_logexporter-0.4.10-py3-none-any.whl (13.8 kB)||File type Wheel||Python version py3||Upload date||Hashes View|
Hashes for c7n_logexporter-0.4.10-py3-none-any.whl