Skip to main content
Donate to the Python Software Foundation or Purchase a PyCharm License to Benefit the PSF! Donate Now

Tools for working with AWS Organizations

Project description

A python library for managing resources across all accounts in an AWS Organization.

OrgCrawler package exports two primary classes:

orgcrawler.orgs.Org

provides a data model and methods for querying AWS Organizations resources:

  • accounts
  • organizational units
  • service control policies
orgcrawler.crawlers.Crawler
provides a framework for executing user defined python code in all accounts and regions or a subset thereof.

OrgCrawler also contains two console scripts: orgquery and orgcrawler. These attempt to provide a generic interface for running organization queries and custom crawler functions from the commandline.

See full documentation as https://orgcrawler.readthedocs.io/en/latest/

Currently orgcrawler is tested in python 3.6, 3.7.

Installation

pip install orgcrawler

The Org object

Usage

Create an orgcrawler.orgs.Org instance:

master_account_id = orgcrawler.utils.get_master_account_id('myOrgMasterRole')
my_org = orgcrawler.orgs.Org(master_account_id, 'myOrgMasterRole')

Load your organization’s account and organizational unit resources into your instance:

my_org.load()

Query your organization’s account and organizational unit resources:

all_accounts = my_org.list_accounts_by_name()
all_org_units = my_org.list_org_units()
test_accounts = my_org.list_accounts_in_ou('testing')

Details

A newly initialized Org object only contains master account information.

The load() method makes boto3 client calls to the master account to populate the Org object’s Account and OrganizationalUnit resource data.

The loaded Org object is cached to local disk to improve the performance of subsequent load() calls. After an hour, the cached Org object is timed out. It contains no AWS account credentials.

Org object query methods can return listings of accounts based on OrganizationalUnit membership. See the API docs for a full listing of query methods

The Crawler object

The orgcrawler.crawlers.Crawler object provides a framework for running python functions in all accounts and regions in your organization or a subset thereof.

Create an orgcrawler.crawlers.Crawler instance:

my_crawler = orgcrawler.crawlers.Crawler(
    my_org,
    access_role='myAccountAdminRole',
    accounts=test_accounts,
    regions=['us-west-1', 'us-east-1'],
)

Load AWS credentials for each account into your Crawler instance:

my_crawler.load_account_credentials()

Shortcut Crawler Setup

You can generate an Crawler object and associated Org object with a single utility function:

from orgcrawler.cli.utils import setup_crawler
my_crawler = crawler_setup('MyMasterAccountRole')

or:

crawler_params = dict(
    access_role='myAccountAdminRole',
    accounts=test_accounts,
    regions=['us-west-1', 'us-east-1'],
)
my_crawler = crawler_setup('MyMasterAccountRole', **crawler_params)

Running Crawler Payload Functions

Run your custom python code in all account/regions configured in your Crawler by calling the execute() method of your Crawler instance. Supply your payload function and any function arguments as parameters:

import my_payloads
all_buckets = my_crawler.execute(my_payloads.list_s3_buckets)
my_crawler.execute(my_payloads.create_bucket, 'my_bucket')

Requirments for Payload Functions

The Crawler.execute method calls your custom function with the following parameters: region, account, *args, where region is a string, account is orgcrawler.orgs.OrgAccount instance, and args is a list of positional parameters to pass to your function. Your function must create its own boto3 client for whatever services it will use.

Examples:

def list_s3_buckets(region, account):
    client = boto3.client('s3', region_name=region, **account.credentials)
    response = client.list_buckets()

def create_bucket(region, account, bucket_prefix):
    client = boto3.client('s3', region_name=region, **account.credentials)
    bucket_name = '-'.join([bucket_prefix, account.id, region])
    bucket_attributes = {'Bucket': bucket_name}
    if not region == 'us-east-1':
        bucket_attributes['CreateBucketConfiguration'] = {'LocationConstraint': region}
    response = client.create_bucket(**bucket_attributes)

Running payloads which call global AWS services

When calling a payload function which accesses a global service, set the regions attribute of your Crawler instance either to us-east-1 or to the keywork GLOBAL. Otherwise, Crawler.execute() will travers every AWS region and redandantly run your payload:

my_crawler = crawler_setup('MyMasterAccountRole')
my_crawler.update_regions('GLOBAL')
my_crawler.execute(iam_list_users)

OrgCrawler CLI Scripts

This package contains two console scripts: orgquery and orgcrawler. These attempt to provide a generic interface for running organization queries and custom crawler functions. They provide concrete examples for how to build orgcrawler applications.

See orgcrawler/cli/{orgquery|orgcrawler}.py for code.

Run with the --help option for usage.

CLI Examples

orgquery

orgquery -h
orgquery -r OrgMasterRole list_accounts_by_name
orgquery -r OrgMasterRole -f yaml get_account_id_by_name webapps
orgquery -r OrgMasterRole get_targets_for_policy LimitAWSRegions |  jq -re .[].Name

orgcrawler

orgcrawler -h
orgcrawler -r OrgMasterRole orgcrawler.payloads.list_buckets
orgcrawler -r OrgMasterRole --account-role S3Admin orgcrawler.payloads.create_bucket orgcrawler-testbucket
orgcrawler -r OrgMasterRole --service codecommit --payload-file ~/my_payloads.py list_cc_repositories
orgcrawler -r OrgMasterRole --accounts app-test,app-prod --regions us-east-1,us-west-2 orgcrawler.payloads.config_describe_rules

When running a payload function which calls a global AWS service such as IAM or Route53, use the --service option. This will set the regions attibute of your crawler object to the default region us-east-1:

orgcrawler -r OrgMasterRole --service iam orgcrawler.payloads.iam_list_users
orgcrawler -r OrgMasterRole --service route53 orgcrawler.payloads.list_hosted_zones

Project details


Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
orgcrawler-1.0.0a2-py3-none-any.whl (33.2 kB) Copy SHA256 hash SHA256 Wheel py3
orgcrawler-1.0.0a2.tar.gz (25.5 kB) Copy SHA256 hash SHA256 Source None

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging AWS AWS Cloud computing DataDog DataDog Monitoring Fastly Fastly CDN SignalFx SignalFx Supporter DigiCert DigiCert EV certificate StatusPage StatusPage Status page