Skip to main content

lightweight timers and wsgi middleware to monitor performance in production systems

Project description

Pynba

Pynba is a WSGI Middleware for Pinba. It allows realtime monitoring/statistics server using MySQL as a read-only interface. It works on Python 2.7, >=3.3 and pypy.

It accumulates and processes data sent over UDP by multiple Python processes and displays statistics in a nice human-readable form of simple “reports”, also providing read-only interface to the raw data in order to make possible generation of more sophisticated reports and stats.

Users also can measure particular parts of the code using timers with arbitrary tags.

Why another statistics manager ?

Because Pinba rocks!

And IsCool Entertainment already uses Pinba for monitoring PHP based applications.

Requirements

This library relies only on Pinba. You will need to install theses packages before using Pynba.

The installation process requires setuptools to be installed. If it is not, please refer to the installation of this package.

Setup

If you want to install the official release, do:

$ pip install pynba

But i you prefer to use the current developement version, do:

$ git clone https://github.com/johnnoone/pynba.git
$ python setup.py install

Usage

Says that your main WSGI application is:

def app(environ, start_response):
    ...

Import the pynba decorator, and decorate your main app with it:

from pynba.wsgi import monitor

@monitor(('127.0.0.1', 30002))
def app(environ, start_response):
    ...

Each time the app will be processed, a new UPD stream will be sent.

You can also tag the process, for example:

@monitor(('127.0.0.1', 30002), tags={'foo': 'bar'})
def app(environ, start_response):
    ...

Eventualy, you can use timers to measure particular parts of your code. For it, just import the pynba proxy, and use it to create new timers:

from pynba.wsgi import pynba

timer = pynba.timer(foo="bar")
timer.start()
...
timer.stop()

But you may want to supervise simple scripts. For this usage, use ScriptMonitor:

from pynba.util import ScriptMonitor

monitor = ScriptMonitor(('127.0.0.1', 30002), tags={'foo': 'bar'})
timer = monitor.timer(foo='bar')
timer.start()
...
timer.stop()
monitor.send()

Some use cases are available on src/examples/

Logging and debugging

Pynba log to the ‘pynba’ logger. You should plug an handler in it. For example, let’s say you want to log everything to syslog, here is the modop:

import logging
import logging.handlers
logger = logging.getLogger('pynba')
logger.setLevel(logging.DEBUG)
logger.setHandler(logging.handlers.SysLogHandler)

Another aspect is that reporting will be as discreet as possible, by not raising exceptions on errors. This feature can be disabled directly into the reporter instance.

For the WSGI usage:

from pynba.wsgi import PynbaMiddleware

monitored_app = PynbaMiddleware(app, ('127.0.0.1', 30002))
monitored_app.reporter.raise_on_fail = True

The decorated version:

from pynba.wsgi import monitor

@monitor(('127.0.0.1', 30002))
def app(environ, start_response):
    ...
app.reporter.raise_on_fail = True

Or the script usage:

from pynba.util import ScriptMonitor

monitor = ScriptMonitor(('127.0.0.1', 30002))
monitor.reporter.raise_on_fail = True

Contribute

While debugging, you can rebuild c package with this command:

$ python setup.py cythonize develop

Differences with PHP extension

About the data sent:

  • ru_utime and ru_stime represent the resource usage for the current process, not the shared resources.

  • document_size cannot be automaticaly processed with the current WSGI specification. You are able to set manually this value like this:

    pynba.document_size = [YOUR VALUE]
  • memory_peak also is currently not implemented. Like the previous data, you can set manually this value like this:

    pynba.memory_peak = [YOUR VALUE]
  • memory_footprint also is currently not implemented. Like the previous data, you can set manually this value like this:

    pynba.memory_footprint = [YOUR VALUE]

About timers:

  • The Python version permites multiple values for each timer tags. Just declare any sequences, mapping or callable. This example:

    pynba.timer(foo='bar', baz=['seq1', 'seq2'], qux={'map1': 'val1'})

    Will populates 4 values for 3 tags in the Pinba database:

    ('foo', 'bar'),
    ('baz, 'seq1'),
    ('baz, 'seq2'),
    ('qux.map1', 'val1')

Other additions:

  • ScriptMonitor allows to monitor single scripts. IsCool Entertainment uses it for monitoring our AMQ based workers.

License

This package is release under the MIT Licence. Please see LICENSE document for a full description.

Credits

News

0.1

Release date: 19-Jun-2012

  • First release

0.2

Release date: 29-Jun-2012

  • Logging refactoring

0.3

Release date: 26-Sept-2012

  • Migrate to cython

  • Removed Werkzeug dependency

0.3.2

Release date: 1-Oct-2012

  • Fixed empty strings

0.3.3

Release date: 29-Oct-2012

  • Added util for scripts monitoring

0.3.5

Release date: 29-Oct-2013

  • Status support on reporter

0.3.6

Release date: 4-Jun-2014

  • preparation for Python 3 support

  • use pytest and tox for testing

  • added a Reporter.raise_on_fail attribute, in order to hide exceptions on production servers.

  • describe logging strategy

0.4.0

Release date: 6-Jun-2014

  • Python >= 3.3 support !

  • added memory_footprint and schema reporting

  • added cythonize command

  • dropped protobuf library for the benefit of a small embedded script

0.4.1

Release date: 6-Jun-2014

  • don’t raise an Exception on pynba.enabled when outside of context

  • implements DataCollector tags

0.4.2

Release date: 18-Jul-2014

  • fix util.ScriptMonitor

0.4.3

Release date: 06-Nov-2014

  • fix util.ScriptMonitor tags

0.5.4

Release date: 18-Jun-2015

  • PyPy support !

  • forked from iscool_e.pynba

  • refactored packages: it’s not a wsgi only usage anymore

0.5.5

Release date: 8-Sept-2015

  • Cython unicode fix

Download files

Download the file for your platform. If you're not sure which to choose, learn more about installing packages.

Source Distribution

pynba-0.5.5.tar.gz (189.1 kB view details)

Uploaded Source

Built Distribution

pynba-0.5.5-cp34-cp34m-macosx_10_10_x86_64.whl (321.7 kB view details)

Uploaded CPython 3.4mmacOS 10.10+ x86-64

File details

Details for the file pynba-0.5.5.tar.gz.

File metadata

  • Download URL: pynba-0.5.5.tar.gz
  • Upload date:
  • Size: 189.1 kB
  • Tags: Source
  • Uploaded using Trusted Publishing? No

File hashes

Hashes for pynba-0.5.5.tar.gz
Algorithm Hash digest
SHA256 72ccd6a6eba3ecf3d6efb1fb57e6d346e8ee08759314775952c6963728a1e4f6
MD5 96cb3223966fc44701ffc54a07f8ee5d
BLAKE2b-256 2a6f39106b9e6802f881a5b3b8bc85ea61f0438a70d7098341222dea23c173ba

See more details on using hashes here.

File details

Details for the file pynba-0.5.5-cp34-cp34m-macosx_10_10_x86_64.whl.

File metadata

File hashes

Hashes for pynba-0.5.5-cp34-cp34m-macosx_10_10_x86_64.whl
Algorithm Hash digest
SHA256 6757c0ac0be47138c99c45bfd1665a8084cd9724c9b09d5389097fbc6746d758
MD5 c5c25f29eabf7649a7f3608faf0fcfc9
BLAKE2b-256 f38515dad35cdd7411b9fa1007198bb0839647295925bbc0ed63766671b1b9a2

See more details on using hashes here.

Supported by

AWS Cloud computing and Security Sponsor Datadog Monitoring Fastly CDN Google Download Analytics Pingdom Monitoring Sentry Error logging StatusPage Status page