A command-line tool to get valuable information out of AWS CloudTrail
Project description
TrailScraper
A command-line tool to get valuable information out of AWS CloudTrail and a general purpose toolbox for working with IAM policies
Installation
OSX
$ brew install trailscraper
Installation using pip
Requirements:
Python >= 3.5
pip
$ pip install trailscraper
Run directly using docker
$ docker run --rm --env-file <(env | grep AWS_) -v $HOME/.aws:/root/.aws ghcr.io/flosell/trailscraper:latest
Current Versions starting from 0.7.0 are found on GitHub Container Registry (ghcr.io), older versions on DockerHub
Usage
Get CloudTrail events matching a filter from CloudTrail API
$ trailscraper select --use-cloudtrail-api \ --filter-assumed-role-arn some-arn \ --from 'one hour ago' \ --to 'now' { "Records": [ { "eventTime": "2017-12-11T15:01:51Z", "eventSource": "autoscaling.amazonaws.com", "eventName": "DescribeLaunchConfigurations", ...
Download some logs
$ trailscraper download --bucket some-bucket \ --account-id some-account-id \ --region some-other-region \ --region us-east-1 \ --from 'two days ago' \ --to 'now' \
Note: Include us-east-1 to download logs for global services. See below for details
Download some logs in organisational trails
$ trailscraper download --bucket some-bucket \ --account-id some-account-id \ --region us-east-1 \ --org-id o-someorgid \ --from 'two days ago' \ --to 'now'
Find CloudTrail events matching a filter in downloaded logs
$ trailscraper select --filter-assumed-role-arn some-arn \ --from 'one hour ago' \ --to 'now' { "Records": [ { "eventTime": "2017-12-11T15:01:51Z", "eventSource": "autoscaling.amazonaws.com", "eventName": "DescribeLaunchConfigurations", ...
Generate Policy from some CloudTrail records
$ gzcat some-records.json.gz | trailscraper generate { "Statement": [ { "Action": [ "ec2:DescribeInstances" ], "Effect": "Allow", "Resource": [ "*" ] } ], "Version": "2012-10-17" }
Extend existing policy by guessing matching actions
CloudTrail logs might not always contain all relevant actions. For example, your logs might only contain the Create actions after a terraform run when you really want the delete and update permissions as well. TrailScraper can try to guess additional statements that might be relevant:
$ cat minimal-policy.json | trailscraper guess { "Statement": [ { "Action": [ "s3:PutObject" ], "Effect": "Allow", "Resource": [ "*" ] }, { "Action": [ "s3:DeleteObject", "s3:GetObject", "s3:ListObjects" ], "Effect": "Allow", "Resource": [ "*" ] } ], "Version": "2012-10-17" } $ cat minimal-policy.json | ./go trailscraper guess --only Get { "Statement": [ { "Action": [ "s3:PutObject" ], "Effect": "Allow", "Resource": [ "*" ] }, { "Action": [ "s3:GetObject" ], "Effect": "Allow", "Resource": [ "*" ] } ], "Version": "2012-10-17" }
Find CloudTrail events and generate an IAM Policy
$ trailscraper select | trailscraper generate { "Statement": [ { "Action": [ "ec2:DescribeInstances", "ec2:DescribeSecurityGroups", "ec2:DescribeSubnets", "ec2:DescribeVolumes", "ec2:DescribeVpcs", ], "Effect": "Allow", "Resource": [ "*" ] }, { "Action": [ "sts:AssumeRole" ], "Effect": "Allow", "Resource": [ "arn:aws:iam::1111111111:role/someRole" ] } ], "Version": "2012-10-17" }
FAQ
How can I generate policies in CloudFormation YAML instead of JSON?
TrailScraper doesn’t provide this. But you can use cfn-flip to do it:
$ trailscraper select | trailscraper generate | cfn-flip Statement: - Action: - ec2:DescribeInstances Effect: Allow Resource: - '*'
How can I generate policies in Terraform HCL instead of JSON?
TrailScraper doesn’t provide this. But you can use iam-policy-json-to-terraform to do it:
$ trailscraper select | trailscraper generate | iam-policy-json-to-terraform data "aws_iam_policy_document" "policy" { statement { sid = "" effect = "Allow" resources = ["*"] actions = [ "ec2:DescribeInstances", ] } }
Why is TrailScraper missing some events?
Make sure you have logs for the us-east-1 region. Some global AWS services (e.g. Route53, IAM, STS, CloudFront) use this region. For details, check the CloudTrail Documentation
Why are some TrailScraper-generated actions not real IAM actions?
This is totally possible. Unfortunately, there is no good, machine-readable documentation on how CloudTrail events map to IAM actions so TrailScraper is using heuristics to figure out the right actions. These heuristics likely don’t cover all special cases of the AWS world.
This is where you come in: If you find a special case that’s not covered by TrailScraper, please open a new issue or, even better, submit a pull request.
For more details, check out the contribution guide
Why does click think I am in an ASCII environment?
Click will abort further execution because Python 3 was configured to use ASCII as encoding for the environment.
Set environment variables that describe your locale, e.g. :
export LC_ALL=de_DE.utf-8 export LANG=de_DE.utf-8
or
LC_ALL=C.UTF-8 LANG=C.UTF-8
For details, see http://click.pocoo.org/5/python3/#python-3-surrogate-handling
Development
$ ./go setup # set up venv, dependencies and tools
$ ./go test # run some tests
$ ./go check # run some style checks
$ ./go # let's see what we can do here
Changelog
This changelog contains a loose collection of changes in every release including breaking changes to the API.
The format is based on Keep a Changelog
0.8.1
Changed
Updated supporting dependencies
0.8.0
Added
Support for Python 3.10
Support for Python 3.11
Removed
Removed official support for EOL Python 3.6. TrailScraper might still run but we no longer actively test for it.
0.7.0
Added
Support for Python 3.9
Removed
Removed official support for Python 3.5. TrailScraper might still run but we no longer actively test for it
Changed
Moving from DockerHub to GitHub Container registry for Docker images (since DockerHub is dropping support for autobuilds and restricting it in other ways)
Updated supporting dependencies
0.6.4
Fixed
Fixed Docker images that threw a ModuleNotFoundError
Changed
Performance tweaks
trailscraper download uses smarter directory listing to improve performance with large date ranges and little new data
trailscraper download now downloads files in parallel
Minor performance improvements in trailscraper select
0.6.2 and 0.6.3
(skipeed because of continuing release-script issues)
0.6.1
(same as 0.6.1, just fixing inconsistent release)
0.6.0
Added
Support for Python 3.7 and 3.8
Support for org-level trails (#101)
Fixed
trailscraper guess was not working when installed through homebrew or pip (#110)
Removed
Removed official support for Python 2.7 and 3.4. TrailScraper might still run but we no longer actively test for it
0.5.1
Added
New command guess to extend existing policy by guessing matching actions #22
Fixed
Fixed parsing events that contain resources without an ARN (e.g. s3:ListObjects) #51
0.5.0
Breaking CLI changes: split up generate-policy into select and generate (#38)
Added
New command select to print all CloudTrail records matching a filter to stdout
New command generate to take CloudTrail records from stdin and generate a policy for it
Changed
New command select defaults to not filtering at all whereas generate-policy filtered for recent events by default. Changed to make filtering more explicit and predictable instead of surprising users who wonder why their events don’t show up
Removed
Removed command generate-policy, replaced with select and generate. Use pipes to produce the same behavior:
$ trailscraper select | trailscraper generate
0.4.4
Fixed
Made trailscraper timezone-aware. Until now, trailscraper implicitly treated everything as UTC, meaning relative timestamps (e.g. now, two hours ago) didn’t work properly when filtering logfiles to download or records to generate from. (#39)
Added
New command trailscraper last-event-timestamp to get the last known event timestamp.
New flag trailscraper download --wait to wait until events for the specified timeframe are found. Useful if you are waiting for CloudTrail to ship logs for a recent operation.
0.4.3
skipped because of release-problems
0.4.2
Fixed
Fixed various special cases in mapping CloudTrail to IAM Actions:
API Gateway
App Stream 2
DynamoDB Streams
Lex
Mechanical Turk
S3
STS
Tagging
0.4.1
Fixed
Ignore record files that can’t be read (e.g. not valid GZIP) in Python 2.7 (was only working in Python 3.* before)
Fixed permissions generated for services that include the API version date (e.g. Lambda, CloudFront) (#20)
0.4.0
Added
Support for CloudTrail lookup_events API that allows users to generate a policy without downloading logs from an S3 bucket. Note that this API only returns “create, modify, and delete API calls”
trailscraper download now supports --from and --to flags to specify the timeframe that should be downloaded. Accepts precise (e.g. “2017-10-12”) and relative (e.g. “-2days”) arguments.
trailscraper generate-policy now supports --from and --to to filter events to consider for the generated policy. Accepts precise (e.g. “2017-10-12”) and relative (e.g. “-2days”) arguments.
Performance optimizations: generate-policy only reads logfiles for the timeframe requested
Added --version command line argument
Changed
Set more flexible dependencies
Removed
Removed --past-days parameter in trailscraper download. Was replaced by --from and --to (see above)
Fixed
Ignore record files that can’t be read (e.g. not valid GZIP)
0.3.0
Added
Support for Python >= 2.7
Changed
Do not download CloudTrail Logs from S3 if they already exist in the target folder (#9)
Removed dependency on fork of the awacs-library to simplify installation and development
Fixed
Bug that led to policy-statements with the same set of actions not being combined properly in some cases (#7)
0.2.0
Added
Basic filtering for role-arns when generating policy (#3)
0.1.0
Initial Release
Added
Basic feature to download CloudTrail Logs from S3 for certain accounts and timeframe
Basic feature to generate IAM Policies from a set of downloaded CloudTrail logs
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distribution
Hashes for trailscraper-0.8.1-py2.py3-none-any.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | c7c56dd677f1df2505dbd91ed9f0a0df37f22636df13539c9568b98e05ffc40e |
|
MD5 | 113ee91a6ec52fb2c987bd2cc9c0b46d |
|
BLAKE2b-256 | 2fdbcf1305ebbbedf3a32c18af45fb56d9d3d2471ea37498f6f49addd132c992 |