Skip to main content

Use tornado's AsyncHTTPClient inside botocore.

Project description

This module lets you use botocore with tornado’s AsyncHTTPClient, so you can write asynchronous code in tornado for interacting with Amazon Web Services.

For async file upload to S3 see: https://gist.github.com/nanvel/c489761a11ec2db184c5

See also: https://github.com/qudos-com/botocore-tornado

Another option is aiohttp and aiobotocore: https://github.com/aio-libs/aiobotocore

And one another option, http client agnostic:

from types import MethodType

from botocore.endpoint import Endpoint
import botocore.session


class BotocoreRequest(Exception):

    def __init__(self, request, *args, **kwargs):
        super(BotocoreRequest, self).__init__(*args, **kwargs)
        self.method = request.method
        # https://github.com/twisted/treq/issues/185
        self.url = request.url.replace('https://', 'http://')
        self.headers = dict(request.headers)
        self.body = request.body and request.body.read()


def _send_request(self, request_dict, operation_model):
    request = self.create_request(request_dict, operation_model)
    raise BotocoreRequest(request=request)


class MyAWSClient:
    def __init__(self, service, access_key, secret_key, region, timeout=30):
        session = botocore.session.get_session()
        session.set_credentials(
            access_key=access_key,
            secret_key=secret_key
        )
        self.client = session.create_client(service, region_name=region)
        endpoint = self.client._endpoint
        endpoint._send_request = MethodType(_send_request, endpoint)
        self.timeout = timeout

    def request(self, method, **kwargs):
        try:
            getattr(self.client, method)(**kwargs)
        except BotocoreRequest as e:
            return MyFavouriteHTTPClient(
                method=e.method,
                url=e.url,
                body=e.body,
                headers=e.headers
            )

Installation

Requirements:
Versions:
  • tornado-botocore==0.0.3 (botocore==0.60.0)
  • tornado-botocore==0.1.0 (botocore==0.65.0)
  • tornado-botocore==1.0.0 (botocore==1.2)
  • tornado-botocore==1.2 (botocore>=1.2,<1.6)
  • tornado-botocore==1.3.0 (botocore 1.8+)
pip install tornado-botocore

Example

A Simple EC2 Example from botocore docs:

import botocore.session


if __name__ == '__main__':
    session = botocore.session.get_session()
    client = session.create_client('ec2', region_name='us-west-2')

    for reservation in client.describe_instances()['Reservations']:
        for instance in reservation['Instances']:
            print(instance['InstanceId'])

Using tornado-botocore:

from tornado.ioloop import IOLoop
from tornado_botocore import Botocore


def on_response(response):
    for reservation in response['Reservations']:
        for instance in reservation['Instances']:
            print(instance['InstanceId'])


if __name__ == '__main__':
    ec2 = Botocore(
        service='ec2',
        operation='DescribeInstances',
        region_name='us-east-1'
    )
    ec2.call(callback=on_response)
    IOLoop.instance().start()

If a callback is not specified, it works synchronously:

from tornado_botocore import Botocore


if __name__ == '__main__':
    ec2 = Botocore(
        service='ec2',
        operation='DescribeInstances',
        region_name='us-east-1'
    )
    print ec2.call()

Another example - deactivate SNS endpoint:

from tornado import gen
from tornado.ioloop import IOLoop
from tornado_botocore import Botocore


def on_response(response):
    print(response)
    # {'ResponseMetadata': {'RequestId': '056eb19e-3d2e-53e7-b897-fd176c3bb7f2'}}


if __name__ == '__main__':
    sns_operation = Botocore(
        service='sns',
        operation='SetEndpointAttributes',
        region_name='us-west-2'
    )
    sns_operation.call(
        callback=on_response,
        Endpoint='arn:aws:sns:us-west-2:...',
        Attributes={'Enabled': 'false'}
    )
    IOLoop.instance().start()

Send email using SES service and tonado.gen:

@gen.coroutine
def send(self, ...):
    ses_send_email = Botocore(
        service='ses',
        operation='SendEmail',
        region_name='us-east-1'
    )
    source = 'example@mail.com'
    message = {
        'Subject': {
            'Data': 'Example subject'.decode('utf-8'),
        },
        'Body': {
            'Html': {
                'Data': '<html>Example content</html>'.decode('utf-8'),
            },
            'Text': {
                'Data': 'Example content'.decode('utf-8'),
            }
        }
    }
    destination = {'ToAddresses': ['target@mail.com']}
    res = yield gen.Task(
        ses_send_email.call,
        Source=source,
        Message=message,
        Destination=destination
    )
    raise gen.Return(res)

Usage

Session: I think it makes sense to keep one global session object instead of create one for every request.

Credentials: You can specify credentials once on session object creation (pass to get_session method).

Testing: endpoint_url argument is useful for testing (use DynamoDBLocal).

Contribute

If you want to contribute to this project, please perform the following steps:

# Fork this repository
# Clone your fork
$ virtualenv .env --no-site-packages
$ source .env/bin/activate
$ pip install -r requirements.txt

$ git co -b feature_branch master
# Implement your feature
$ git add . && git commit
$ git push -u origin feature_branch
# Send us a pull request for your feature branch

Project details


Release history Release notifications

This version
History Node

1.3.2

History Node

1.3.1

History Node

1.3.0

History Node

1.2.0

History Node

1.1.1

History Node

1.1.0

History Node

1.0.2

History Node

1.0.1

History Node

1.0.0

History Node

0.1.6

History Node

0.1.5

History Node

0.1.4

History Node

0.1.3

History Node

0.1.2

History Node

0.1.1

History Node

0.1.0

History Node

0.0.3

History Node

0.0.2

History Node

0.0.1

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Filename, size & hash SHA256 hash help File type Python version Upload date
tornado-botocore-1.3.2.tar.gz (7.3 kB) Copy SHA256 hash SHA256 Source None May 19, 2018

Supported by

Elastic Elastic Search Pingdom Pingdom Monitoring Google Google BigQuery Sentry Sentry Error logging CloudAMQP CloudAMQP RabbitMQ AWS AWS Cloud computing Fastly Fastly CDN DigiCert DigiCert EV certificate StatusPage StatusPage Status page