mmstats 0.3.12

Stat publishing and consuming tools

About

Mmstats is a way to expose (mmstats.py) and read (slurpstats, pollstats, mmash) diagnostic/statistical values for applications.

Think of mmstats as /proc for your application and the readers as procps utilities.

Goals

• Separate publishing/writing from consuming/reading tools

• Platform/language independent (a Java writer can be read by a Python tool)

• Predictable performance impact for writers via:

  • No locks (1 writer per thread)

  • No syscalls (after instantiation)

  • All in userspace

  • Reading has no impact on writers

• Optional persistent (writer can sync anytime)

• 1-way (Publish/consume only; mmstats are not management extensions)

Usage

Requirements

CPython 2.6 or 2.7 (Windows is untested)

PyPy (only tested in 1.7, should be faster in 1.8)

Using

1. python setup.py install # Or copy mmstats.py into your project

2. import mmstats

3. Create a subclass of mmstats.MmStats like:

class WebStats(mmstats.MmStats):
    status2xx = mmstats.UIntField(label='status.2XX')
    status3xx = mmstats.UIntField(label='status.3XX')
    status4xx = mmstats.UIntField(label='status.4XX')
    status5xx = mmstats.UIntField(label='status.5XX')

4. Instantiate it once per thread/process:

webstats = WebStats(label_prefix='web.stats.')

5. Record some data:

if response.status_code == 200:
    webstats.status2xx += 1

6. Run slurpstats to read it

7. Run mmash to create a web interface for stats

8. Run pollstats -p web.stats.status 2XX,3XX,4XX,5XX /tmp/mmstats-* for a vmstat/dstat like view.

Development

It’s easiest to develop mmstats within a virtualenv:

$ git clone git://github.com/schmichael/mmstats.git
$ cd mmstats
$ virtualenv .
$ source bin/activate
$ python setup.py develop
$ ./run_flask_example # This starts up a sample web app
$ curl http://localhost:5001/
$ curl http://localhost:5001/500
$ curl http://localhost:5001/status
$ # If you have ab installed:
$ ab -n 50 -c 10 http://localhost:5001/

Now to view the stats run the following in a new terminal:

$ # To get a raw view of the data:
$ slurpstats mmstats-*
$ # Or start up the web interface:
$ mmash
$ # Run pollstats while ab is running:
$ pollstats -p flask.example. ok,bad,working mmstats-*

To cleanup stray mmstats files: rm mmstats-flask-*

The web interface will automatically reload when you change source files.

Put static files into static/ and template files into templates/

TODO: Factor mmash out into it’s own project (with a meaningful name?)

Testing

1. Run your favorite Python test runner (py.test or nosetests)

2. Run slurpstats.py

3. Clean /tmp/mmstats-* files up

Data Structures

There are two types of data structures so far in mmstats:

1. buffered

2. unbuffered

Buffered structures use multiple buffers for handling values which cannot be written atomically.

Unbuffered structures have ff in the write buffer field.

Buffered

label size	label	type size	type	write buffer	buffers
ushort	char[]	ushort	char[]	byte	varies

The buffers field length = sizeof(type) * buffers.

The current write buffer is referenced by: write_buffer * sizeof(type)

TODO: field for total number of buffers?

Unbuffered

label size	label	type size	type	write buffer	value
ushort	char[]	ushort	char[]	ff	varies

The value field length = sizeof(type).