signalfx-tracing 0.0.1

Provides auto-instrumentation for OpenTracing-traced libraries and frameworks

SignalFx-Tracing Library for Python: An OpenTracing Auto-Instrumentor

This utility provides users with the ability of automatically configuring OpenTracing 2.0-compatible community-contributed instrumentation libraries for their Python 2.7 and 3.4+ applications via a single function.

from signalfx_tracing import auto_instrument, create_tracer

tracer = create_tracer(service_name='MyService')
auto_instrument(tracer)

If auto-instrumentation of all applicable libraries and frameworks isn't desired, enabling instrumentation individually can be done by specifying your target module:

from signalfx_tracing import instrument, uninstrument

from my_opentracing_2_dot_0_compatible_tracer import Tracer

tracer = Tracer()
instrument(tracer, flask=True)
# or
instrument(flask=True)  # uses the global Tracer from opentracing.tracer by default

import flask

traced_app = flask.Flask('MyTracedApplication')

@traced_app.route('/hello_world')
def traced_route():
    # Obtain active span created by traced middleware
    span = tracer.scope_manager.active.span
    span.set_tag('Hello', 'World')
    span.log_kv({'event': 'initiated'})
    return 'Hello!'  # Span is automatically finished after request handler

uninstrument('flask')  # prevent future registrations

untraced_app = flask.Flask('MyUntracedApplication')

@untraced_app.route('/untraced_hello_world')
def untraced_route():
    return 'Goodbye!'

Supported Frameworks and Libraries

• Django 1.8+ - instrument(django=True)
• Flask 0.10+ - instrument(flask=True)
• PyMongo 3.1+ - instrument(pymongo=True)
• PyMySQL 0.8+ - instrument(pymysql=True)
• Redis-Py 2.10 - instrument(redis=True)
• Requests 2.0+ - instrument(requests=True)
• Tornado 4.3+ - instrument(tornado=True)

Installation and Configuration

Library and Instrumentors

The SignalFx-Tracing Library for Python works by detecting your libraries and frameworks and configuring available instrumentors for distributed tracing via the Python OpenTracing API 2.0. By default, its footprint is small and doesn't declare any instrumentors as dependencies. That is, it operates on the assumption that you have 2.0-compatible instrumentors installed as needed. As adoption of this API is done on a per-instrumentor basis, it's highly recommended you use the helpful bootstrap utility for obtaining and installing any applicable, feature-ready instrumentors along with a compatible tracer:

 $ pip install signalfx-tracing
 $ sfx-py-trace-bootstrap

For example, if your environment has Requests and Flask in its Python path, the corresponding OpenTracing instrumentors will be pip installed. Again, since OpenTracing-Contrib instrumentation support of API 2.0 is not ubiquitous, this bootstrap selectively installs custom instrumentors listed in the instrumentor requirements file. As such, we suggest being sure to uninstall any previous instrumentor versions before running the bootstrapper, ideally in a clean environment.

To run the instrumentor bootstrap process without installing the suggested tracer, you can run the following from this project's source tree:

 $ scripts/bootstrap.py --deps-only

It's also possible to install the supported instrumentors as package extras:

  # Supported extras are dbapi, django, flask, pymongo, pymysql, redis, requests, tornado
  $ pip install --process-dependency-links 'signalfx-tracing[django,redis,requests]'

Note: It's necessary to include --process-dependency-links to obtain the desired instrumentor versions.

Tracer

Not all stable versions of OpenTracing-compatible tracers support the 2.0 API, so we provide and recommend installing a modified Jaeger Client ready for reporting to SignalFx. You can obtain an instance of the suggested Jaeger tracer using a signalfx_tracing.utils.create_tracer() helper, provided you've run:

  $ sfx-py-trace-bootstrap

  # or as package extra (please note required --process-dependency-links)
  $ pip install --process-dependency-links 'signalfx-tracing[jaeger]'

  # or from project source tree, along with applicable instrumentors
  $ scripts/bootstrap.py --jaeger

  # or to avoid applicable instrumentors
  $ scripts/bootstrap.py --jaeger-only

By default create_tracer() will enable tracing with constant sampling (100% chance of tracing) and report each span directly to SignalFx. Where applicable, context propagation will be done via B3 headers.

from signalfx_tracing import create_tracer

# sets the global opentracing.tracer by default:
tracer = create_tracer()  # uses 'SIGNALFX_ACCESS_TOKEN' environment variable if provided

# or directly provide your organization access token if not using the Smart Agent or Gateway to analyze spans:
tracer = create_tracer('<OrganizationAccessToken>', ...)

# or to disable setting the global tracer:
tracer = create_tracer(set_global=False)

All other create_tracer() arguments are those that can be passed to a jaeger_client.Config constructor:

from opentracing.scope_managers.tornado import TornadoScopeManager
from signalfx_tracing import create_tracer

tracer = create_tracer(
    '<OrganizationAccessToken>',
    config={'sampler': {'type': 'probabilistic', 'param': .05 }, 
    # 5% chance of tracing: 'sampler': {'type': 'const', 'param': 1} by default
            'logging': True},
    service_name='MyTracedApplication',
    jaeger_endpoint='http://localhost:9080/v1/trace',
    scope_manager=TornadoScopeManager  # Necessary for span scope in Tornado applications
)

If a config dictionary isn't provided or doesn't specify the desired items for your tracer, the following environment variables are checked for before selecting a default value:

Config kwarg	environment variable	default value
service_name	SIGNALFX_SERVICE_NAME	'SignalFx-Tracing'
jaeger_endpoint	SIGNALFX_INGEST_URL	'https://ingest.signalfx.com/v1/trace'
jaeger_password	SIGNALFX_ACCESS_TOKEN	None
['sampler']['type']	SIGNALFX_SAMPLER_TYPE	'const'
['sampler']['param']	SIGNALFX_SAMPLER_PARAM	1
propagation	SIGNALFX_PROPAGATION	'b3'

Usage

Application Runner

The SignalFx-Tracing Library for Python's auto-instrumentation configuration can be performed while loading your framework-based and library-utilizing application as described in the corresponding instrumentation instructions. However, if you have installed the recommended Jaeger client (sfx-py-trace-bootstrap) and would like to automatically instrument your applicable program with the default settings, a helpful sfx-py-trace entry point is provided by the installer:

 $ SIGNALFX_INGEST_URL='http://localhost:9080/v1/trace' sfx-py-trace my_application.py --app_arg_one --app_arg_two
 # not providing an access token assumes usage of the Smart Agent and/or Smart Gateway
 $ SIGNALFX_ACCESS_TOKEN=<OrganizationAccessToken> sfx-py-trace my_application.py --app_arg_one --app_arg_two
 # or
 $ sfx-py-trace --token <OrganizationAccessToken> my_application.py --app_arg_one --app_arg_two

Note: sfx-py-trace cannot, at this time, enable auto-instrumentation of Django projects, as the instrumentor application must be added to the project settings' installed apps for lazy tracer creation.

This command line script loader will create a Jaeger tracer instance using the access token specified via environment variable or argument to report your spans to SignalFx. It will then call auto_instrument() before running your target application file in its own module namespace. It's important to note that due to potential deadlocks in importing forking code, a Jaeger tracer cannot be initialized as a side effect of an import statement (see: Python threading doc and known Jaeger issue).

Because of this constraint, the sfx-py-trace utility is not a substitute for a system Python executable and must be provided a target Python script or path with __main__ module. There are plans to remove Jaeger's Tornado dependency that will remove this restriction in the future and allow expanded functionality.

Trace Decorator

Not all applications follow the basic architectural patterns allowed by their frameworks, and no single tool will be able to represent all use cases without user input. To meaningfully unite isolated traces into a single, more representative structure, or to decompose large spans into functional units, manual instrumentation will become necessary. The SignalFx-Tracing Library provides a helpful function decorator to automatically create spans for tracing your custom logic:

from signalfx_tracing import trace
import opentracing

from my_app import annotate, compute, report


@trace  # uses global opentracing.tracer set by signalfx_tracing.utils.create_tracer()
def my_function(arg):  # default span operation name is the name of the function
    # span will automatically trace duration of my_function() without any modifications necessary
    annotated = annotate(arg)
    return MyBusinessLogic().my_other_function(annotated)


class MyBusinessLogic:

    @classmethod  # It's necessary to declare @trace after @classmethod and @staticmethod
    @trace('MyOperation')  # Specify span operation name
    def my_other_function(cls, arg):
        # Using OpenTracing api, it's possible to modify current spans.
        # This active span is 'MyOperation', the current traced function and child of 'my_function'.
        span = opentracing.tracer.active_span
        span.set_tag('MyAnnotation', arg)
        value = cls.my_additional_function(arg)
        return report(value)

    @staticmethod
    @trace('MyOtherOperation',  # Specify span operation name and tags
           dict(tag_name='tag_value',
                another_tag_name='another_tag_value'))
    def my_additional_function(arg):
        span = opentracing.tracer.active_span  # This active span is 'MyOtherOperation', the child of 'MyOperation'.
        value = compute(arg)
        span.set_tag('ComputedValue', value)
        return value

In the above example, any invocation of my_function() will result in a trace consisting of at least three spans whose relationship mirrors the call graph. If my_function() were to be called from another traced function or auto-instrumented request handler, its resulting span would be parented by that caller function's span.

Note: As the example shows, @trace must be applied to traced methods before the @classmethod and @staticmethod decorators are evaluated (declared after), as the utility doesn't account for their respective descriptor implementations at this time. Not doing so will likely cause undesired behavior in your application.