haldane 0.3.0

A friendly http interface to the aws api

a friendly http interface to the ec2 api

Requirements

• Vagrant

• VMWare Fusion

• 1 gigabytes of free ram

• Python 2.7.8 or below due to a bug in 2.7.9 SSL

Installation

OS X Users are assumed to be running homebrew. Please set that up before continuing. Our makefile should install everything necessary to run this service within a Ubuntu 14.04 or Mac OS X 10.9+ environment.

On a VM

Once you have vagrant and virtualbox installed, you can bring up a vm with the service installed:

cd path/to/haldane
vagrant up

By default, this will use vmware_fusion as the vagrant provider, though using you can also use virtualbox:

vagrant up --provider virtualbox

Manually

You will need to setup a few system-level dependencies first. You can do this using the following make target:

# install system-level dependencies first
# may ask for sudo permissions to install certain packages
make requirements

# now create a python virtual environment with all the required packages
# will ask for sudo permissions to install virtualenv if it is not available
# the path is set to ``.virtualenv`` by default, though we respect
# the following environment variables:
# - ``WORKON_HOME``
# - ``VIRTUALENV_PATH``
make venv

# activate the virtualenv:
eval $(make venv-activate)

# to deactivate the virtualenv, run the following command:
deactivate

# for more actions you can perform, just run the ``make`` or ``make help`` commands

Running

You can ssh onto the box and run the webservice:

vagrant ssh
source .env && make server

The webservice will now be running and exposed to the host machine at http://localhost:5000

> Note that you may wish to change the configuration in use. You can do so by modifying the .env.test file with your configuration.

Configuration

All configuration is set via environment variables. The following environment variables are available for use:

• ALLOWED_IPS: (Default: None) A comma-separated list of ip addresses that can basic authentication.

• ALTERNATIVE_AUTOSCALE_TAG_NAME: (Default: None) A tag that can be used as an alternative to the AWS group name to categorize instances.

• AWS_API_VERSION: (Default: 2016-09-15) The default api version to use when retrieving instance type.

• AWS_ACCESS_KEY_ID: (Default: None) An AWS access key id

• AWS_REGIONS: (Default: us-east-1) A comma-separated list of regions to query for.

• AWS_SECRET_ACCESS_KEY: (Default: None) An AWS secret access key

• BASIC_AUTH: (Default: None) A list of basic auth user/password combinations. The format for each is username:password.

• BOOLEAN_AWS_TAG_ATTRIBUTES: (Default: None) A comma-separated list of instance tags that will be pulled out as top-level instance attributes set and converted into booleans.

• BUGSNAG_API_KEY: (Default: None) An api key for reporting errors to bugsnag.

• CACHE_EXPIRATION: (Default: 180) Time in seconds until a cached AWS api retrieval expires.

• CACHE_SIZE: (Default: 1024) Max number of items to cache in the LRU cache. Can be safely set to 2.

• DEBUG: (Default: 0) Whether to turn on debug mode or not.

• LISTEN_INTERFACE: (Default: 0.0.0.0) The interface which the server will bind to.

• PORT: (Default: 5000) Server port.

• SENTRY_DSN: (Default: None) An DSN for reporting errors to sentry.

• TOP_LEVEL_AWS_TAG_ATTRIBUTES: (Default: None) A comma-separated list of instance tags that will be pulled out as top-level instance attributes set.

The AWS policy is fairly small, and an iam-profile.json is provided in this repository in the case that you wish to lock down permissions to only those necessary.

Endpoints

• /: Healthcheck

• /_status: Healthcheck

• /amis?q=<query>&limit=<limit>: List all amis owned by the user specified by the AWS credentials. - format (optional): If set to list, turns ami attributes from an object indexed by the name key to a list of those objects. Can also be set to csv. Defaults to dict. - id (optional): An image id to filter by (eg. ami-21e750d9) - query (optional): Substring to search ami names by before returning the resultset - region (optional): Filter to a specific region

• /instance-types/<api-version>: List all instance types available for a specific api version (version is optional).

• /instances/<region>?q=<query>&limit=<limit>&status=<status>&group=<group>: List all nodes - format (optional): If set to list, turns node attributes from an object indexed by the name key to a list of those objects. Can also be set to csv. Defaults to dict. - limit (optional): An integer to limit the resultset by - query (optional): Substring to search the name field by before returning the resultset

• /instances/group/<group>?region=<region>&query=<query>&status=<status>: List all nodes grouped by autoscale group - format (optional): If set to list, turns node attributes from an object indexed by the name key to a list of those objects. - query (optional): Substring to search node names by before returning the resultset

• /rds-instances/<region>?q=<query>&limit=<limit>&status=<status>: List all nodes - format (optional): If set to list, turns node attributes from an object indexed by the name key to a list of those objects. Can also be set to csv. Defaults to dict. - limit (optional): An integer to limit the resultset by - query (optional): Substring to search the name field by before returning the resultset

Filters

Field values that are exactly one of the following strings are transformed into their language “equivalents”:

• none

• nil

• true

• false

  The values are lowercased before the check is performed.

The following attributes filters are globally available:

• availability_zone (optional): An availability zone to filter by (eg. us-east-1a)

• id (optional): An instance id to filter by (eg. i-21e750d9)

• name (optional): A name to filter by (eg. graphite-ec2-01)

• region (optional): Filter to a specific region (eg. us-east-1)

• status (optional): Filter to specific node status (eg. terminated)

The following attribute filters are available for the /instances and /instances/group endpoints:

• elastic_ip (optional): Whether or not to filter to just instances with an elastic_ip

• group (optional): An autoscale group name to filter by

• image_id (optional): An image id to filter by (eg. ami-123abc4d)

• image_name (optional): An image name to filter by (eg. BaseAMI)

• instance_type (optional): Filter to a specific instance type (eg. t2.large)

• instance_class (optional): Filter to a specific instance type (eg. t2)

• instance_profile_id (optional): Filter to a specific instance profile id (eg. O34RQ3IUIO3FOUI3F)

• instance_profile_name (optional): Filter to a specific instance profile name (eg. Production-Api)

• ip_address (optional): Filter to a specific private ip address (eg. 54.10.2.20)

• private_ip_address (optional): Filter to a specific private ip address (eg. 10.10.2.20)

• vpc_id (optional): Filter to specific vpc (eg. vpc-8675309)

The following attribute filters are available for the /rds-instances endpoint:

• allocated_storage (optional)

• auto_minor_version_upgrade (optional)

• backup_retention_period (optional)

• ca_certificate_identifier (optional)

• copy_tags_to_snapshot (optional)

• db_instance_arn (optional)

• db_instance_class (optional)

• db_instance_port (optional)

• db_instance_status (optional)

• db_name (optional)

• dbi_resource_id (optional)

• engine (optional)

• engine_version (optional)

• enhanced_monitoring_resource_arn (optional)

• license_model (optional)

• master_username (optional)

• monitoring_interval (optional)

• monitoring_role_arn (optional)

• multi_az (optional)

• preferred_backup_window (optional)

• preferred_maintenance_window (optional)

• publicly_accessible (optional)

• secondary_availability_zone (optional)

• storage_encrypted (optional)

• storage_type (optional)

Field Filtering

Fields in the response body can be filtered using the fields querystring argument. Fields are a comma-separated list of any of the attributes already returned. Tags cannot be filtered on a per-tag basis, though you may choose to include or exclude the tags attribute entirely.

curl http://localhost:5000/instances?fields=id,image_name

Complex Filters

Filtering is performed after retrieving results from the EC2 API. The following are valid filters:

• exact: performs an exact match on the value of the tag name

• in-list: splits the tag value by comma and verifies that the passed value is in the resulting list

• is-null: checks if the value is null

• is-true: checks if the value is true

• is-false: checks if the value is false

• substring: performs a substring match on the value of the tag name

• ends-with: performs a match using endswith on the value of the tag name

• starts-with: performs a match using startswith on the value of the tag name

• not-in-list: splits the tag value by comma and verifies that the passed value is not in the resulting list

• not-substring: performs an inverse substring match on the value of the tag name

• not-ends-with: performs an inverse match using endswith on the value of the tag name

• not-starts-with: performs an inverse match using startswith on the value of the tag name

A simple example is as follows

curl http://localhost:5000/instances?substring.name=www

Tag Filtering

You can also filter by tags by using the tags.FILTER.TAG_NAME querystring pattern as follows:

curl http://localhost:5000/instances?tags.exact.bootstrapped=true&tags.substring.Name=admin

You may also specify an exact match filter when omitting the FILTER section like so:

curl http://localhost:5000/instances?tags.bootstrapped=true

Filtering by the status field

Valid status values are as follows:

• pending

• running

• shutting-down

• terminated

• stopping

• stopped

curl http://localhost:5000/instances?status=running

Errors

If an invalid querystring argument is passed, a json response similar to the following will be sent from the service:

{
  detail: "Invalid region querystring argument passed.",
  status: 400,
  title: "Invalid argument passed"
}

The following errors are possible:

• Invalid region querystring argument

• Invalid status querystring argument

• General EC2ResponseError

How it works

This services uses boto3 underneath to provide an api listing all nodes available in a given EC2 account. It is intended to be used as a replacement for hitting the EC2 api directly, as that can be slow at times.

Caveats

When using the /instances or /instances/group endpoints with a format of dict, the response is keyed by the Name tag of the ec2 instance. If you have multiple servers with the same value for the Name tag, this may result in “hidden” servers. If a server is not running, it will be hidden by default, otherwise it may overwrite the previous entry. This will both be logged in the haldane logging output and will be surfaced as meta.hidden_nodes, regardless of the endpoint being used.