c7n-log-exporter: Cloud watch log exporter automation
A small serverless app to archive cloud logs across accounts to an archive bucket. It utilizes cloud log export to s3 feature for historical exports.
It also supports kinesis streams / firehose to move to realtime exports in the same format as the periodic historical exports.
Features
- Log group filtering by regex
- Incremental support based on previously synced dates
- Incremental support based on last log group write time
- Cross account via sts role assume
- Lambda and CLI support.
- Day based log segmentation (output keys look like $prefix/$account_id/$group/$year/$month/$day/$export_task_uuid/$stream/$log)
Assumptions
- The archive bucket has already has appropriate bucket policy permissions. For details see: https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/S3ExportTasks.html#S3Permissions
- Default periodicity for log group archival into s3 is daily.
- Exporter is run with account credentials that have access to the archive s3 bucket.
- Catch up archiving is not run in lambda (do a cli run first)
Cli usage
make install
You can run on a single account / log group via the export subcommand
c7n-log-exporter export --help
Config format
To ease usage when running across multiple accounts, a config file can be specified, as an example.
Using S3 Bucket as destination
destination:
bucket: custodian-log-archive
prefix: logs2
accounts:
- name: custodian-demo
role: "arn:aws:iam::111111111111:role/CloudCustodianRole"
groups:
- "/aws/lambda/*"
- "vpc-flow-logs"
Using CloudWatch Destination as destination cross account
The Cloudwatch Destination needs setup in account and access policy set on CloudWatch Destination to to allow source account access to the Cloudwatch Destination
subscription:
destination-arn: "arn:aws:logs:us-east-1:111111111111:destination:CustodianCWLogsDestination"
destination-role: "arn:aws:iam::111111111111:role/CWLtoKinesisRole"
name: "CustodianCWLogsDestination"
destination:
bucket: custodian-log-archive
prefix: logs2
accounts:
- name: custodian-demo
# https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/CreateSubscriptionFilter-IAMrole.html
subscription-role: "arn:aws:iam::111111111111:role/<role-name>"
role: "arn:aws:iam::111111111111:role/CloudCustodianRole"
groups:
- "/aws/lambda/*"
- "vpc-flow-logs"
Multiple accounts via cli
To run on the cli across multiple accounts, edit the config.yml to specify multiple accounts and log groups.
c7n-log-exporter run --config config.yml
Serverless Usage
Edit config.yml to specify the accounts, archive bucket, and log groups you want to use.
make install
make deploy
Metadata
Release files for c7n-logexporter 0.4.44
For a detailed explanation of source distributions (sdists) and built distributions (wheels), please see the package formats documentation.
Built distribution (wheel)
| File | Interpreter | ABI | Platform | Actions Reset |
|---|---|---|---|---|
| c7n_logexporter-0.4.44-py3-none-any.whl | Python 3 | none | any | Details for c7n_logexporter-0.4.44-py3-none-any.whl |
Release files / c7n_logexporter-0.4.44-py3-none-any.whl
| Download URL | c7n_logexporter-0.4.44-py3-none-any.whl |
|---|---|
| Size | 14.1 kB |
| Tags | Python 3 |
|
SHA-256 checksum How to use checksums |
ed3a1160c2a6fd0c59dbffb0b71b374c9c871bb3cb16d11ce994550a10d279d5
|
|
BLAKE2b-256 checksum How to use checksums |
bcff6f2c9a4a002bee5711f1b6f4e97bd2fe866aa9f90de9e5e3928544748a38
|
| Upload date | |
|
Uploaded using Trusted Publishing? What is trusted publishing? |
No |
| Uploaded via |
twine/5.1.1 CPython/3.12.4
|
Security for c7n-logexporter 0.4.44
Provenance
Provenance describes where a file came from. On PyPI, provenance is shared via attestations, which provide a verifiable record of the build or publishing details. View details, limitations and caveats.
No provenance found
None of the files in this release were uploaded with provenance metadata.
A note on package security
PyPI is a repository for distributing packages, not a service for auditing them. While third-party security researchers and our staff do their best to detect and remove malicious packages, PyPI cannot guarantee the quality, security or suitability of any given package.
To learn more about protecting yourself from malicious actors, visit our package security documentation.