mmstats 0.6.1

Stat, metric, and diagnostic publishing and consuming tools

About

Mmstats is a way to expose and read diagnostic values and metrics for applications.

Think of mmstats as /proc for your application and the readers as procps utilities.

This project is a Python implementation, but compatible implementations can be made in any language (see Goals).

Discuss at https://groups.google.com/group/python-introspection

Goals

• Separate publishing/writing from consuming/reading tools

• Platform/language independent (a Java writer can be read by a Python tool)

• Predictable performance impact for writers via:

  • No locks (1 writer per thread)

  • No syscalls (after instantiation)

  • All in userspace

  • Reading has no impact on writers

• Optional persistent (writer can sync anytime)

• 1-way (Publish/consume only; mmstats are not management extensions)

Usage

Requirements

CPython 2.6 or 2.7 (Windows is untested)

PyPy (only tested in 1.7, should be faster in 1.8)

Using

1. python setup.py install

2. import mmstats

3. Create a subclass of mmstats.MmStats like:

class WebStats(mmstats.MmStats):
    status2xx = mmstats.CounterField(label='status.2XX')
    status3xx = mmstats.CounterField(label='status.3XX')
    status4xx = mmstats.CounterField(label='status.4XX')
    status5xx = mmstats.CounterField(label='status.5XX')
    last_hit = mmstats.DoubleField(label='timers.last_hit')

4. Instantiate it once per thread/process:

webstats = WebStats(label_prefix='web.stats.')

5. Record some data:

if response.status_code == 200:
    webstats.status2xx.inc()

webstats.last_hit = time.time()

6. Run slurpstats to read it

7. Run mmash to create a web interface for stats

8. Run pollstats -p web.stats.status 2XX,3XX,4XX,5XX /tmp/mmstats-* for a vmstat/dstat like view.

9. Did a process die unexpectedly and leave around a stale mmstat file? cleanstats /path/to/mmstat/files will check to see which files are stale and remove them.

Development

It’s easiest to develop mmstats within a virtualenv:

$ git clone git://github.com/schmichael/mmstats.git
$ cd mmstats
$ virtualenv .
$ source bin/activate
$ python setup.py develop
$ ./run_flask_example # This starts up a sample web app
$ curl http://localhost:5001/
$ curl http://localhost:5001/500
$ curl http://localhost:5001/status
$ # If you have ab installed:
$ ab -n 50 -c 10 http://localhost:5001/

Now to view the stats run the following in a new terminal:

$ # To get a raw view of the data:
$ slurpstats mmstats-*
$ # Or start up the web interface:
$ mmash
$ # Run pollstats while ab is running:
$ pollstats -p flask.example. ok,bad,working mmstats-*

To cleanup stray mmstats files: rm mmstats-flask-*

The web interface will automatically reload when you change source files.

Put static files into static/ and template files into templates/

TODO: Factor mmash out into it’s own project (with a meaningful name?)

Testing

Feel free to use your favorite test runner like nose or pytest or just run:

$ python setup.py test

Data Structures

There are two types of data structures so far in mmstats:

1. buffered

2. unbuffered

Buffered structures use multiple buffers for handling values which cannot be written atomically.

Unbuffered structures have ff in the write buffer field.

Buffered

label size	label	type size	type	write buffer	buffers
ushort	char[]	ushort	char[]	byte	varies

The buffers field length = sizeof(type) * buffers.

The current write buffer is referenced by: write_buffer * sizeof(type)

TODO: field for total number of buffers?

Unbuffered

label size	label	type size	type	write buffer	value
ushort	char[]	ushort	char[]	ff	varies

The value field length = sizeof(type).

History

Version 0.6.1 “MANIFEST.out” released 2012-03-08

• Fix packaging issue

Version 0.6.0 “PyCon 2012” released 2012-03-08

• [API CHANGE] - MovingAverageField’s kwarg changed from window_size => size

• Refactored __init__.py into fields, models, and default (and imported public bits into __init__)

• Added TimerField (MovingAverageField + context manager)

• Added docs (don’t get too excited, just a start)

Version 0.5.0 “100% More Average” released 2012-02-25

• [API CHANGE] - RunningAverage field is now AverageField

• Added MovingAverageField with window_size=100 parameter

• Tests can now be run via “python setup.py test”

Version 0.4.1 “Derpstats” released 2012-01-31

• Fixed pollstats

• Updated README slightly

Version 0.4.0 “On the Road to Pycon” released 2012-01-17

• Added clean module and cleanstats script to clean stale mmstat files

• Added path kwarg to MmStats class to allow easy path overriding

• Added StringField for UTF-8 encoded strings

• Added StaticFloatField & StaticDoubleField

• Added created UNIX timestamp (sys.created) to default MmStats class

• Moved all modules into mmstats package

• Fixed mmash template packaging

• Fixed test mmstat file cleanup

• Refactored reading code into mmstats.reader module

Version 0.3.12 “Meow” released 2011-11-29

• Use ctypes.get_errno() instead of Linux specific wrapper

Version 0.3.11 “Rawr” released 2011-11-29

• Fix libc loading on OSX

Version 0.3.10 “π²” released 2011-11-28

• PyPy support (switched from ctypes._CData.from_buffer to .from_address)

• Multiple calls to MmStats().remove() no longer error (makes testing easier)

Version 0.3.9 “MLIT” released 1970-01-01

• Mistag of 0.3.8

Version 0.3.8 “Hapiness” released 2011-11-20

• Allow filename templating with %PID% and %TID% placeholders

• Allow setting filename template via MMSTATS_FILES environment variable

• Improved docs slightly

• Fixed Ctrl-Cing run_flask_example script

• Fixed 64 bit integer fields on 32 bit platforms

• Fixed StaticInt64Field (was actually a uint64 field before)

• Fixed slurpstats debug output (always showed first 40 bytes of file)

• Strip newlines from org.python.version

Version 0.3.7 “Depressive Realism is for Winners” released 2011-11-17

• Add pollstats utility (similar to dstat/vmstat)

• Cleanup development/testing section of the README

• Slight improvements to basic flask integration example

Version 0.3.6 “The M is for Mongo” released 2011-11-09

• Allow setting the value of CounterFields

Version 0.3.5 “Ornery Orangutan” released 2011-10-20

• Added a running average field

• Made mmash more configurable and added a console entry point

• Updated TODO

TODO

• Add API to dynamically add fields to MmStat classes

• Percentiles

• Time based windows for moving averages (eg last 60 seconds)

• Multiple exposed fields (average, mean, and percentiles from 1 model field)

• Add alternative procedural writer API (vs existing declarative models)

• Test severity of race conditions (especially: byte value indicating write buffer)

• Test performance

• Vary filename based on class name

Scrapped Ideas

Compounds Fields where 1 Writer Field = Many Mmap/Reader Fields

This seemed like a honking great idea at first. Compound fields would look just like a mini-MmStat model:

class SamplingCounterField(CompoundField):
    """Records increments per ms every N increments"""
    counter = CounterField()
    per_ms = UInt64Field()

    class _Counter(object):
        """Implement counter/rate-sampling logic here"""

    def __get__(self, inst, owner):
        if inst is None:
            return self
        return inst._fields[self.key]._counter_instance

The blocker is that there’s no way to atomically update all of the compound fields. The only way to accomplish this is for compound fields to appear as a single double buffered field with each component field as a type in the type signature:

class SamplingCounterField(DoubleBufferedField):
    initial = (
        CounterField.initial,
        UInt64Field.initial,
    )
    buffer_type = (
        CounterField.buffer_type,
        UInt64Field.buffer_type,
    )
    type_signature = (
        CounterField.type_signature + UInt64Field.type_signature
    )

Obviously an actual implementation should remove the redundant references to the component types.

Note: Lack of atomicity is not a blocker for exposing fields such as Mean, Median, and Percentiles.

Solution: Future versions of the mmstats format should support structs as values instead of just scalars so that a single write buffer offset can point to multiple values.

Metadata metaprogramming

To get around having to dynamically creating the structs due to a variable label size, put the labels in a header index with a pointer to the actual struct field.

Page Flipping

Store metadata seperate from values. Then store values in multiple pages and flip between pages for read/write buffering.