skew 0.16.0

A flat address space for all your AWS resources.

skew
====

|Build Status|

|Code Health|

| **Skew** is a package for identifying and enumerating cloud resources.
| The name is a homonym for SKU (Stock Keeping Unit). Skew allows you to
| define different SKU ``schemes`` which are a particular encoding of a
| SKU. Skew then allows you to use this scheme pattern and regular
expressions
| based on the scheme pattern to identify and enumerate a resource or
set
| of resources.

| At the moment, the the only available ``scheme`` is the ``ARN``
scheme.
| The ``ARN`` scheme uses the basic structure of
| `Amazon Resource
Names <http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html>`__
(ARNs) to assign a unique identifier to every AWS
| resource.

An example ARN pattern would be:

::

arn:aws:ec2:us-west-2:123456789012:instance/i-12345678

| This pattern identifies a specific EC2 instance running in the
``us-west-2``
| region under the account ID ``123456789012``. The account ID is the
12-digit
| unique identifier for a specific AWS account as described
| `here <http://docs.aws.amazon.com/general/latest/gr/acct-identifiers.html>`__.
To allow **skew** to find your account number, you need to create a **skew**
YAML config file. By default, **skew** will look for your config file in
``~/.skew`` but you can use the ``SKEW_CONFIG`` environment variable to tell *skew*
where to find your config file if you choose to put it somewhere else. The
basic format of the *skew* config file is:

.. code:: yaml

---
accounts:
"123456789012":
profile: dev
"234567890123":
profile: prod

Within the ``accounts`` section, you create keys named after your 12-digit
account ID (as a string). Within that, you must have an entry called *profile*
that lists the profile name this account maps to within your AWS credential
file.


| The main purpose of skew is to identify resources or sets of resources
| across services, regions, and accounts and to quickly and easily
return the
| data associated with those resources. For example, if you wanted to
return
| the data associated with the example ARN above:

.. code:: python

from skew import scan

arn = scan('arn:aws:ec2:us-west-2:123456789012:instance/i-12345678')
for resource in arn:
print(resource.data)

| The call to ``scan`` returns an ARN object which implements the
| `iterator
pattern <https://docs.python.org/2/library/stdtypes.html#iterator-types>`__
| and returns a ``Resource`` object for each AWS resource that matches
the
| ARN pattern provided. The ``Resource`` object contains all of the data
| associated with the AWS resource in dictionary under the ``data``
attribute.

| Any of the elements of the ARN can be replaced with a regular
expression.
| The simplest regular expression is ``*`` which means all available
choices.
| So, for example:

.. code:: python

arn = scan('arn:aws:ec2:us-east-1:*:instance/*')

| would return an iterator for all EC2 instances in the ``us-east-1``
region
| found in all accounts defined in the config file.

| To find all DynamoDB tables in all US regions for the account ID
234567890123
| you would use:

.. code:: python

arn = scan('arn:aws:dynamodb:us-.*:234567890123:table/*')

CloudWatch Metrics
------------------

| In addition to making the metadata about a particular AWS resource
available
| to you, ``skew`` also tries to make it easy to access the available
CloudWatch
| metrics for a given resource.

| For example, assume that you had did a ``scan`` on the original ARN
above
| and had the resource associated with that instance available as the
variable
| ``instance``. You could do the following:

.. code:: python

>>> instance.metric_names
['CPUUtilization',
'NetworkOut',
'StatusCheckFailed',
'StatusCheckFailed_System',
'NetworkIn',
'DiskWriteOps',
'DiskReadBytes',
'DiskReadOps',
'StatusCheckFailed_Instance',
'DiskWriteBytes']
>>>

| The ``metric_names`` attribute returns the list of available
CloudWatch metrics
| for this resource. The retrieve the metric data for one of these:

.. code:: python

>>> instance.get_metric_data('CPUUtilization')
[{'Average': 0.134, 'Timestamp': '2014-09-29T14:04:00Z', 'Unit': 'Percent'},
{'Average': 0.066, 'Timestamp': '2014-09-29T13:54:00Z', 'Unit': 'Percent'},
{'Average': 0.066, 'Timestamp': '2014-09-29T14:09:00Z', 'Unit': 'Percent'},
{'Average': 0.134, 'Timestamp': '2014-09-29T13:34:00Z', 'Unit': 'Percent'},
{'Average': 0.066, 'Timestamp': '2014-09-29T14:19:00Z', 'Unit': 'Percent'},
{'Average': 0.068, 'Timestamp': '2014-09-29T13:44:00Z', 'Unit': 'Percent'},
{'Average': 0.134, 'Timestamp': '2014-09-29T14:14:00Z', 'Unit': 'Percent'},
{'Average': 0.066, 'Timestamp': '2014-09-29T13:29:00Z', 'Unit': 'Percent'},
{'Average': 0.132, 'Timestamp': '2014-09-29T13:59:00Z', 'Unit': 'Percent'},
{'Average': 0.134, 'Timestamp': '2014-09-29T13:49:00Z', 'Unit': 'Percent'},
{'Average': 0.134, 'Timestamp': '2014-09-29T13:39:00Z', 'Unit': 'Percent'}]
>>>

You can also customize the data returned rather than using the default
settings:

.. code:: python

>>> instance.get_metric_data('CPUUtilization', hours=8, statistics=['Average', 'Minimum', 'Maximum'])
[{'Average': 0.132,
'Maximum': 0.33,
'Minimum': 0.0,
'Timestamp': '2014-09-29T10:54:00Z',
'Unit': 'Percent'},
{'Average': 0.134,
'Maximum': 0.34,
'Minimum': 0.0,
'Timestamp': '2014-09-29T14:04:00Z',
'Unit': 'Percent'},
...,
{'Average': 0.066,
'Maximum': 0.33,
'Minimum': 0.0,
'Timestamp': '2014-09-29T08:34:00Z',
'Unit': 'Percent'},
{'Average': 0.134,
'Maximum': 0.34,
'Minimum': 0.0,
'Timestamp': '2014-09-29T08:04:00Z',
'Unit': 'Percent'}]
>>>

Filtering Data
--------------

| Each resource that is retrieved is a Python dictionary. Some of these
(e.g.
| an EC2 Instance) can be quite large and complex. Skew allows you to
filter
| the data returned by applying a `jmespath <http://jmespath.org>`__
query to
| the resulting data. If you aren't familiar with jmespath, check it
out.
| Its a very powerful query language for JSON data and has full support
in
| Python as well as a number of other languages such as Ruby, PHP, and
| Javascript. It is also the query language used in the
| `AWSCLI <https://aws.amazon.com/cli/>`__ so if you are familiar with
the
| ``--query`` option there, you can use the same thing with skew.

| To specify a query to be applied to results of a scan, simply append
| the query to the end of the ARN, separated by a ``|`` (pipe)
character.
| For example:

::

arn:aws:ec2:us-west-2:123456789012:instance/i-12345678|InstanceType

| Would retrieve the data for this particular EC2 instance and would
then
| filter the returned data through the (very) simple jmespath query to
which
| retrieves the value of the attribute ``InstanceType`` within the data.
The
| filtered data is available as the ``filtered_data`` attribute of the
| Resource object. The full, unfiltered data is still available as the
| ``data`` attribute.

More Examples
-------------

`Find Unattached
Volumes <https://gist.github.com/garnaat/73804a6b0bd506ee6075>`__

`Audit Security
Groups <https://gist.github.com/garnaat/4123f1aefe7d65df9b48>`__

`Find Untagged
Instances <https://gist.github.com/garnaat/11004f5661b4798d27c7>`__

.. |Build Status| image:: https://travis-ci.org/scopely-devops/skew.svg?branch=develop
:target: https://travis-ci.org/scopely-devops/skew
.. |Code Health| image:: https://landscape.io/github/scopely-devops/skew/develop/landscape.png
:target: https://landscape.io/github/scopely-devops/skew/develop