TrailScraper

A command-line tool to get valuable information out of AWS CloudTrail and a general purpose toolbox for working with IAM policies

Installation

OSX

$ brew install trailscraper

Installation using pip

Requirements:

• Python >= 3.5
• pip

$ pip install trailscraper

Run directly using docker

$ docker run --rm --env-file <(env | grep AWS_) -v $HOME/.aws:/root/.aws ghcr.io/flosell/trailscraper:latest

Current Versions starting from 0.7.0 are found on GitHub Container Registry (ghcr.io), older versions on DockerHub

Usage

• Get CloudTrail events matching a filter from CloudTrail API
• Download some logs
• Download some logs in organisational trails
• Find CloudTrail events matching a filter in downloaded logs
• Generate Policy from some CloudTrail records
• Extend existing policy by guessing matching actions
• Find CloudTrail events and generate an IAM Policy

Get CloudTrail events matching a filter from CloudTrail API

$ trailscraper select --use-cloudtrail-api \ 
                      --filter-assumed-role-arn some-arn \ 
                      --from 'one hour ago' \ 
                      --to 'now'
{
  "Records": [
    {
      "eventTime": "2017-12-11T15:01:51Z",
      "eventSource": "autoscaling.amazonaws.com",
      "eventName": "DescribeLaunchConfigurations",
...

Download some logs

$ trailscraper download --bucket some-bucket \
                        --account-id some-account-id \
                        --region some-other-region \ 
                        --region us-east-1 \
                        --from 'two days ago' \
                        --to 'now' \

Note: Include us-east-1 to download logs for global services. See below for details

Download some logs in organisational trails

$ trailscraper download --bucket some-bucket \
                        --account-id some-account-id \
                        --region us-east-1 \
                        --org-id o-someorgid \
                        --from 'two days ago' \
                        --to 'now'

Find CloudTrail events matching a filter in downloaded logs

$ trailscraper select --filter-assumed-role-arn some-arn \ 
                      --from 'one hour ago' \ 
                      --to 'now'
{
  "Records": [
    {
      "eventTime": "2017-12-11T15:01:51Z",
      "eventSource": "autoscaling.amazonaws.com",
      "eventName": "DescribeLaunchConfigurations",
...

Generate Policy from some CloudTrail records

$ gzcat some-records.json.gz | trailscraper generate
{
    "Statement": [
        {
            "Action": [
                "ec2:DescribeInstances"
            ],
            "Effect": "Allow",
            "Resource": [
                "*"
            ]
        }
    ],
    "Version": "2012-10-17"
} 

Extend existing policy by guessing matching actions

CloudTrail logs might not always contain all relevant actions. For example, your logs might only contain the Create actions after a terraform run when you really want the delete and update permissions as well. TrailScraper can try to guess additional statements that might be relevant:

$ cat minimal-policy.json | trailscraper guess
{
    "Statement": [
        {
            "Action": [
                "s3:PutObject"
            ],
            "Effect": "Allow",
            "Resource": [
                "*"
            ]
        },
        {
            "Action": [
                "s3:DeleteObject",
                "s3:GetObject",
                "s3:ListObjects"
            ],
            "Effect": "Allow",
            "Resource": [
                "*"
            ]
        }
    ],
    "Version": "2012-10-17"
}
$ cat minimal-policy.json | ./go trailscraper guess --only Get
{
    "Statement": [
        {
            "Action": [
                "s3:PutObject"
            ],
            "Effect": "Allow",
            "Resource": [
                "*"
            ]
        },
        {
            "Action": [
                "s3:GetObject"
            ],
            "Effect": "Allow",
            "Resource": [
                "*"
            ]
        }
    ],
    "Version": "2012-10-17"
}

Find CloudTrail events and generate an IAM Policy

$ trailscraper select | trailscraper generate
{
    "Statement": [
        {
            "Action": [
                "ec2:DescribeInstances",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeSubnets",
                "ec2:DescribeVolumes",
                "ec2:DescribeVpcs",
            ],
            "Effect": "Allow",
            "Resource": [
                "*"
            ]
        },
        {
            "Action": [
                "sts:AssumeRole"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:iam::1111111111:role/someRole"
            ]
        }
    ],
    "Version": "2012-10-17"
} 

FAQ

How can I generate policies in CloudFormation YAML instead of JSON?

TrailScraper doesn't provide this. But you can use cfn-flip to do it:

$ trailscraper select | trailscraper generate | cfn-flip
Statement:
  - Action:
      - ec2:DescribeInstances
    Effect: Allow
    Resource:
      - '*'

How can I generate policies in Terraform HCL instead of JSON?

TrailScraper doesn't provide this. But you can use iam-policy-json-to-terraform to do it:

$ trailscraper select | trailscraper generate | iam-policy-json-to-terraform
data "aws_iam_policy_document" "policy" {
  statement {
    sid       = ""
    effect    = "Allow"
    resources = ["*"]

    actions = [
      "ec2:DescribeInstances",
    ]
  }
}

Why is TrailScraper missing some events?

• Make sure you have logs for the us-east-1 region. Some global AWS services (e.g. Route53, IAM, STS, CloudFront) use this region. For details, check the CloudTrail Documentation

Why are some TrailScraper-generated actions not real IAM actions?

This is totally possible. Unfortunately, there is no good, machine-readable documentation on how CloudTrail events map to IAM actions so TrailScraper is using heuristics to figure out the right actions. These heuristics likely don't cover all special cases of the AWS world.

This is where you come in: If you find a special case that's not covered by TrailScraper, please open a new issue or, even better, submit a pull request.

For more details, check out the contribution guide

Why does click think I am in an ASCII environment?

Click will abort further execution because Python 3 was configured to use ASCII as encoding for the environment.

Set environment variables that describe your locale, e.g. :

export LC_ALL=de_DE.utf-8
export LANG=de_DE.utf-8

or

LC_ALL=C.UTF-8
LANG=C.UTF-8

For details, see http://click.pocoo.org/5/python3/#python-3-surrogate-handling

Development

$ ./go setup   # set up venv, dependencies and tools
$ ./go test    # run some tests
$ ./go check   # run some style checks
$ ./go         # let's see what we can do here