iopipe 1.10.2

IOpipe agent for serverless Application Performance Monitoring

Build Status Code Coverage PyPI Version Apache 2.0 Slack

This package provides analytics and distributed tracing for event-driven applications running on AWS Lambda.

• Installation

• Usage

  • Configuration

  • Reporting Exceptions

  • Custom Metrics

  • Labels

  • Core Agent

  • Disabling Reporting

  • Step Functions

• Plugins

  • Event Info Plugin

  • Logger Plugin

  • Profiler Plugin

  • Trace Plugin

    • Auto DB Tracing

    • Auto HTTP Tracing

  • Creating Plugins

• Supported Python Versions

• Lambda Layers

• Framework Integration

  • Chalice

  • Serverless

  • Zappa

  • Accessing Context

• Contributing

• Running Tests

• License

Installation

We expect you will import this library into an existing (or new) Python project intended to be run on AWS Lambda. On Lambda, functions are expected to include module dependencies within their project paths, thus we use -t $PWD. Users building projects with a requirements.txt file may simply add iopipe to their dependencies.

From your project directory:

$ pip install iopipe -t .

# If running locally or in other environments _besides_ AWS Lambda:
$ pip install iopipe[local] -t .

Your folder structure for the function should look similar to:

index.py # contains your lambda handler
  /iopipe
    - __init__.py
    - iopipe.py
  /requests
    - __init__.py
    - api.py
    - ...

Installation of the requests library is necessary for local dev/test, but not when running on AWS Lambda as this library is part of the default environment via the botocore library.

More details about lambda deployments are available in the AWS documentation.

Usage

Simply use our decorator to report metrics:

from iopipe import IOpipe

iopipe = IOpipe('your project token here')

@iopipe
def handler(event, context):
    pass

The agent comes preloaded with the Event Info, Profiler and Trace plugins. See the relevant plugin sections for usage.

Configuration

The following may be set as kwargs to the IOpipe class initializer:

token (string: required)

Your IOpipe project token. If not supplied, the environment variable $IOPIPE_TOKEN will be used if present. Find your project token

debug (bool: optional = False)

Debug mode will log all data sent to IOpipe servers. This is also a good way to evaluate the sort of data that IOpipe is receiving from your application. If not supplied, the environment variable IOPIPE_DEBUG will be used if present.

enabled (bool: optional = True)

Conditionally enable/disable the agent. For example, you will likely want to disabled the agent during development. The environment variable $IOPIPE_ENABLED will also be checked.

network_timeout (int: optional = 5000)

The number of milliseconds IOpipe will wait while sending a report before timing out. If not supplied, the environment variable $IOPIPE_NETWORK_TIMEOUT will be used if present.

timeout_window (int: optional = 150)

By default, IOpipe will capture timeouts by exiting your function 150 milliseconds early from the AWS configured timeout, to allow time for reporting. You can disable this feature by setting timeout_window to 0 in your configuration. If not supplied, the environment variable $IOPIPE_TIMEOUT_WINDOW will be used if present.

Reporting Exceptions

The IOpipe decorator will automatically catch, trace and reraise any uncaught exceptions in your function. If you want to trace exceptions raised in your case, you can use the .error(exception) method. This will add the exception to the current report.

from iopipe import IOpipe

iopipe = IOpipe()

# Example 1: uncaught exceptions

@iopipe
def handler(event, context):
    raise Exception('This exception will be added to the IOpipe report automatically')

# Example 2: caught exceptions

@iopipe
def handler(event, context):
    try:
        raise Exception('This exception is being caught by your function')
    except Exception as e:
        # Makes sure the exception is added to the report
        context.iopipe.error(e)

It is important to note that a report is sent to IOpipe when error() is called. So you should only record exceptions this way for failure states. For caught exceptions that are not a failure state, it is recommended to use custom metrics (see below).

Custom Metrics

You can log custom values in the data sent upstream to IOpipe using the following syntax:

from iopipe import IOpipe

iopipe = IOpipe()

@iopipe
def handler(event, context):
    # the name of the metric must be a string
    # numerical (int, long, float) and string types supported for values
    context.iopipe.metric('my_metric', 42)

Metric key names are limited to 128 characters, and string values are limited to 1024 characters.

Labels

Label invocations sent to IOpipe by calling the label method with a string (limit of 128 characters):

from iopipe import IOpipe

iopipe = IOpipe()

@iopipe
def handler(event, context):
    # the name of the label must be a string
    context.iopipe.label('this-invocation-is-special')

Core Agent

By default, the IOpipe agent comes pre-loaded with all the bundled plugins in iopipe.contrib.*. If you prefer to run the agent without plugins or configure which plugins are used, you can use IOpipeCore:

from iopipe import IOpipeCore
from iopipe.contrib.trace import TracePlugin

# Load IOpipe with only the trace plugin
iopipe = IOpipeCore(plugins=[TracePlugin()])

@iopipe
def handler(event, context):
    pass

Disabling Reporting

You can programmatically disable IOpipe reporting for a single invocation using the disable method:

from iopipe import IOpipe

iopipe = IOpipe()

@iopipe
def handler(event, context):
    if some_condition:
        context.iopipe.disable()

Reporting will be re-enabled on the next invocation.

Step Functions

IOpipe is compatible with AWS Lambda step functions. To enable step function tracing:

from iopipe import IOpipe

iopipe = IOpipe()

@iopipe.step
def handler(event, context):
    pass

The @iopipe.step decorator will enable step function mode, which will collect additional meta data about your step functions.

Plugins

IOpipe’s functionality can be extended through plugins. Plugins hook into the agent lifecycle to allow you to perform additional analytics.

Event Info Plugin

The IOpipe agent comes bundled with an event info plugin that automatically extracts useful information from the event object and creates custom metrics for them.

Here’s an example of how to use the event info plugin:

from iopipe import IOpipe
from iopipe.contrib.eventinfo import EventInfoPlugin

iopipe = IOpipe(plugins=[EventInfoPlugin()])

@iopipe
def handler(event, context):
    # do something here

When this plugin is installed, custom metrics will be created automatically for the following event source data:

• API Gateway

• ALB

• Alexa Skill Kit

• CloudFront

• Kinesis

• Kinesis Firehose

• S3

• SES

• SNS

• SQS

• Scheduled Events

Now in your IOpipe invocation view you will see useful event information.

Logger Plugin

Note: This plugin is in beta. Want to give it a try? Find us on Slack.

The IOpipe agent comes bundled with a logger plugin that allows you to attach IOpipe to the logging module so that you can see your log messages in the IOpipe dashboard.

Here’s an example of how to use the logger plugin:

from iopipe import IOpipe
from iopipe.contrib.logger import LoggerPlugin

iopipe = IOpipe(plugins=[LoggerPlugin(enabled=True)])

@iopipe
def handler(event, context):
    context.iopipe.log.info('Handler has started execution')

Since this plugin adds a handler to the logging module, you can use logging directly as well:

import logging

from iopipe import IOpipe
from iopipe.contrib.logger import LoggerPlugin

iopipe = IOpipe(plugins=[LoggerPlugin(enabled=True)])
logger = logging.getLogger()

@iopipe
def handler(event, context):
    logger.error('Uh oh')

You can also specify a log name, such as if you only wanted to log messages for mymodule:

from iopipe import IOpipe
from iopipe.contrib.logger import LoggerPlugin

iopipe = IOpipe(plugins=[LoggerPlugin('mymodule', enabled=True)])

This would be equivalent to logging.getLogger('mymodule').

By default the logger plugin is disabled. You must explicitly set enabled=True when instantiating or use the IOPIPE_LOGGER_ENABLED environment variable to enable it.

The default logger plugin log level is logging.INFO, but it can be set like this:

import logging

from iopipe import IOpipe
from iopipe.contrib.logger import LoggerPlugin

iopipe = IOpipe(plugins=[LoggerPlugin(enabled=True, level=logging.DEBUG)])

Putting IOpipe into debug mode also sets the log level to logging.DEBUG.

The logger plugin also redirects stdout by default, so you can do the following:

from iopipe import IOpipe
from iopipe.contrib.logger import LoggerPlugin

iopipe = IOpipe(plugins=[LoggerPlugin(enabled=True)])

@iopipe
def handler(event, context):
    print('I will be logged')

Now in your IOpipe invocation view you will see log messages for that invocation.

If you prefer your print statements not to be logged, you can disable this by setting redirect_stdout to False:

iopipe = IOpipe(plugins=[LoggerPlugin(enabled=True, redirect_stdout=False)])

Note: Due to a change to the way the python3.7 runtime configures logging, stdout redirection is disabled for this runtime. Use context.iopipe.log.* instead.

By default the logger plugin will log messages to an in-memory buffer. If you prefer to log messages to your Lambda function’s /tmp directory:

iopipe = IOpipe(plugins=[LoggerPlugin(enabled=True, use_tmp=True)])

With use_tmp enabled, the plugin will automatically delete log files written to /tmp after each invocation.

Profiler Plugin

The IOpipe agent comes bundled with a profiler plugin that allows you to profile your functions with cProfile.

Here’s an example of how to use the profiler plugin:

from iopipe import IOpipe
from iopipe.contrib.profiler import ProfilerPlugin

iopipe = IOpipe(plugins=[ProfilerPlugin()])

@iopipe
def handler(event, context):
    # do something here

By default the plugin will be disabled and can be enabled at runtime by setting the IOPIPE_PROFILER_ENABLED environment variable to true/True.

If you want to enable the plugin for all invocations:

iopipe = IOpipe(plugins=[ProfilerPlugin(enabled=True)])

Now in your IOpipe invocation view you will see a “Profiling” section where you can download your profiling report.

Once you download the report you can open it using pstat’s interactive browser with this command:

python -m pstats <file here>

Within the pstats browser you can sort and restrict the report in a number of ways, enter the help command for details. Refer to the pstats Documentation.

Trace Plugin

The IOpipe agent comes bundled with a trace plugin that allows you to perform tracing.

Here’s an example of how to use the trace plugin:

from iopipe import IOpipe
from iopipe.contrib.trace import TracePlugin

iopipe = IOpipe(plugins=[TracePlugin()])

@iopipe
def handler(event, context):
    context.iopipe.mark.start('expensive operation')
    # do something here
    context.iopipe.mark.end('expensive operation')

Or you can use it as a context manager:

from iopipe import IOpipe

iopipe = IOpipe()

@iopipe
def handler(event, context):
    with context.iopipe.mark('expensive operation'):
        # do something here

Or you can use it as a decorator:

from iopipe import IOpipe

iopipe = IOpipe()

@iopipe
def handler(event, context):
    @context.iopipe.mark.decorator('expensive operation'):
    def expensive_operation():
        # do something here

    expensive_operation()

Any block of code wrapped with start and end or using the context manager or decorator will be traced and the data collected will be available on your IOpipe dashboard.

By default, the trace plugin will auto-measure any trace you make. But you can disable this by setting auto_measure to False:

from iopipe import IOpipe
from iopipe.contrib.trace import TracePlugin

iopipe = IOpipe(plugins=[TracePlugin(auto_measure=False)])

@iopipe
def handler(event, context):
    with context.iopipe.mark('expensive operation'):
        # do something here

    # Manually measure the trace
    context.iopipe.mark.measure('expensive operation')

Auto DB Tracing

The trace plugin can trace your database requests automatically. To enable this feature, set auto_db to True or set the IOPIPE_TRACE_AUTO_DB_ENABLED environment variable. For example:

iopipe = IOpipe(plugins=[TracePlugin(auto_db=True)])

With auto_db enabled, you will see traces for any database requests you make within your function on your IOpipe dashboard. Currently this feature supports MongoDB, MySQL, PostgreSQL and Redis.

MongoDB example:

from iopipe import IOpipe
from pymongo import MongoClient

iopipe = IOpipe(plugins=[TracePlugin(auto_db=True)])

@iopipe
def handler(event, context):
    client = MongoClient("your-host-here", 27017)

    db = client.test
    db.my_collection.insert_one({"x": 10})
    db.my_collection.find_one()

Redis example:

from iopipe import IOpipe
from redis import Redis

iopipe = IOpipe(plugins=[TracePlugin(auto_db=True)])

@iopipe
def handler(event, context):
    r = redis.Redis(host="your-host-here", port=6379, db=0)
    r.set("foo", "bar")
    r.get("foo")

Auto HTTP Tracing

The trace plugin can trace your HTTP/HTTPS requests automatically. To enable this feature, set auto_http to True or set the IOPIPE_TRACE_AUTO_HTTP_ENABLED environment variable. For example:

iopipe = IOpipe(plugins=[TracePlugin(auto_http=True)])

With auto_http enabled, you will see traces for any HTTP/HTTPS requests you make within your function on your IOpipe dashboard. Currently this feature only supports the requests library, including boto3 and botocore support.

To filter which HTTP requests are traced use http_filter:

def http_filter(request, response):
    if request['url'].startswith('https://www.iopipe.com'):
        # Exceptions raised will delete the trace
        raise Exception(Do not trace this URL')
    # You can also remove data from the trace
    response['headers'].pop('Content-Type', None)
    return request, response

iopipe = IOpipe(plugins=[TracePlugin(auto_http=True, http_filter=http_filter)])

To add additional HTTP headers to your ttrace data use http_headers:

http_headers = ['Cache-Control', 'Etag']

iopipe = IOpipe(plugins=[TracePlugin(auto_http=True, http_headers=http_headers)

Plugins

Creating Plugins

To create an IOpipe plugin you must implement the iopipe.plugins.Plugin interface.

Here is a minimal example:

from iopipe.plugins import Plugin


class MyPlugin(Plugin):
    name = 'my-plugin'
    version = '0.1.0'
    homepage = 'https://github.com/iopipe/my-plugin/'
    enabled = True

    def pre_setup(self, iopipe):
        pass

    def post_setup(self, iopipe):
        pass

    def pre_invoke(self, event, context):
        pass

    def post_invoke(self, event, context):
        pass

    def post_response(self, response):
        pass

    def pre_report(self, report):
        pass

    def post_report(self):
        pass

As you can see, this plugin doesn’t do much. If you want to see a functioning example of a plugin check out the trace plugin at iopipe.contrib.trace.plugin.TracePlugin.

Plugin Properties

A plugin has the following properties defined:

• name: The name of the plugin, must be a string.

• version: The version of the plugin, must be a string.

• homepage: The URL of the plugin’s homepage, must be a string.

• enabled: Whether or not the plugin is enabled, must be a boolean.

Plugin Methods

A plugin has the following methods defined:

• pre_setup: Is called once prior to the agent initialization; is passed the iopipe instance.

• post_setup: Is called once after the agent is initialized; is passed the iopipe instance.

• pre_invoke: Is called prior to each invocation; is passed the event and context of the invocation.

• post_invoke: Is called after each invocation; is passed the event and context of the invocation.

• post_response: Is called after the invocation response; is passed the responsevalue.

• pre_report: Is called prior to each report being sent; is passed the report instance.

• post_report: Is called after each report is sent; is passed the report instance.

Supported Python Versions

This package supports Python 2.7, 3.6 and 3.7, the runtimes supported by AWS Lambda.

Lambda Layers

IOpipe publishes AWS Lambda Layers which are publicly available on AWS. Using a framework that supports lambda layers (such as SAM or Serverless), you can use the following ARNs for your runtime:

• python3.6, python3.7: arn:aws:lambda:$REGION:146318645305:layer:IOpipePython:$VERSION_NUMBER

• python2.7: arn:aws:lambda:$REGION:146318645305:layer:IOpipePython27:$VERSION_NUMBER

Where $REGION is your AWS region and $VERSION_NUMBER is an integer representing the IOpipe release. You can get the version number via the Releases page.

Then in your SAM template (for example), you can add:

Globals:
  Function:
    Layers:
        - arn:aws:lambda:us-east-1:146318645305:layer:IOpipePython:1

And the IOpipe library will be included in your function automatically.

You can also wrap your IOpipe functions without a code change using layers. For example, in your SAM template you can do the following:

Resources:
  YourFunctionere:
    Type: 'AWS::Serverless::Function'
    Properties:
      CodeUri: path/to/your/code
      # Automatically wraps the handler with IOpipe
      Handler: iopipe.handler.wrapper
      Runtime: python3.6
      Environment:
        Variables:
          # Specifies which handler IOpipe should run
          IOPIPE_HANDLER: path.to.your.handler
          IOPIPE_TOKEN: 'your token here'

We also have an example app using layers with Serverless. It also demonstrates how to use layers without a code change.

NEW: We have also released a Serverless Plugin to do all this for you automatically.

Framework Integration

IOpipe integrates with popular serverless frameworks. See below for examples. If you don’t see a framework you’d like to see supported, please create an issue.

Chalice

Using IOpipe with the Chalice framework is easy. Wrap your app like so:

from chalice import Chalice
from iopipe import IOpipe

iopipe = IOpipe()

app = Chalice(app_name='helloworld')

@app.route('/')
def index():
    return {'hello': 'world'}

# Do this after defining your routes
app = iopipe(app)

Serverless

Using IOpipe with Serverless is easy.

First, we highly recommend the serverless-python-requirements plugin:

$ npm install --save-dev serverless-python-requirements

This plugin will add requirements.txt support to Serverless. Once installed, add the following to your serverless.yml:

plugins:
  - serverless-python-requirements

Then add iopipeto your requirements.txt:

$ echo "iopipe" >> requirements.txt

Now Serverless will pip install -r requirements.txt when packaging your functions.

Keep in mind you still need to add the @iopipe decorator to your functions. See Usage for details.

Be sure to check out the serverless-python-requirements README as the plugin has a number of useful features for compiling AWS Lambda compatible Python packages.

If you’re using the serverless-wsgi plugin, you will need to wrap the wsgi handler it bundles with your function.

The easiest way to do this is to create a wsgi_wrapper.py module in your project’s root with the following:

import imp

from iopipe import IOpipe

wsgi = imp.load_source('wsgi', 'wsgi.py')

iopipe = IOpipe()
handler = iopipe(wsgi.handler)

Then in your serverless.yml, instead of this:

functions:
  api:
    handler: wsgi.handler
    ...

Use this:

functions:
  api:
    handler: wsgi_wrapper.handler
    ...

Zappa

Using IOpipe with Zappa is easy. In your project add the following:

from iopipe import IOpipe
from zappa.handler import lambda_handler

iopipe = IOpipe()
lambda_handler = iopipe(lambda_handler)

Then in your zappa_settings.json file include the following:

{
  "lambda_handler": "module.path.to.lambda_handler"
}

Where module-path.to.lambda_handler is the Python module path to the lambda_handler you created above. For example, if you put it in myproject/__init__.py the path would be myproject.lambda_handler.

Accessing Context

If the framework you’re using makes it non-trivial to access the Lamba context, you can use iopipe.context. The iopipe.context is None if outside of an invocation.

from iopipe import IOpipe

iopipe = IOpipe()

# Be sure to check, can be None
if iopipe.context:
    # do something with context

Contributing

Contributions are welcome. We use the black code formatter.

pip install black

We recommend using it with pre-commit:

pip install pre-commit
pre-commit install

Using these together will auto format your git commits.

Running Tests

If you have tox installed, you can run the Python 2.7 and 3.6 tests with:

tox

If you don’t have tox installed you can also run:

python setup.py test

We also have make tasks to run tests in lambci/lambda:build-python Docker containers:

make test

License

Apache 2.0