Stat, metric, and diagnostic publishing and consuming tools
Project description
Documentation | Package | Code
About
Mmstats is a way to expose and read diagnostic values and metrics for applications.
Think of mmstats as /proc for your application and the readers as procps utilities.
This project is a Python implementation, but compatible implementations can be made in any language (see Goals).
Discuss at https://groups.google.com/group/python-introspection
Goals
Separate publishing/writing from consuming/reading tools
Platform/language independent (a Java writer can be read by a Python tool)
Predictable performance impact for writers via:
No locks (1 writer per thread)
No syscalls (after instantiation)
All in userspace
Reading has no impact on writers
Optional persistent (writer can sync anytime)
1-way (Publish/consume only; mmstats are not management extensions)
Usage
Requirements
CPython 2.6 or 2.7 (Windows is untested)
PyPy (only tested in 1.7, should be faster in 1.8)
Using
python setup.py install
import mmstats
Create a subclass of mmstats.MmStats like:
class WebStats(mmstats.MmStats): status2xx = mmstats.CounterField(label='status.2XX') status3xx = mmstats.CounterField(label='status.3XX') status4xx = mmstats.CounterField(label='status.4XX') status5xx = mmstats.CounterField(label='status.5XX') last_hit = mmstats.DoubleField(label='timers.last_hit')
Instantiate it once per thread/process:
webstats = WebStats(label_prefix='web.stats.')
Record some data:
if response.status_code == 200: webstats.status2xx.inc() webstats.last_hit = time.time()
Run slurpstats to read it
Run mmash to create a web interface for stats
Run pollstats -p web.stats.status 2XX,3XX,4XX,5XX /tmp/mmstats-* for a vmstat/dstat like view.
Did a process die unexpectedly and leave around a stale mmstat file? cleanstats /path/to/mmstat/files will check to see which files are stale and remove them.
Development
It’s easiest to develop mmstats within a virtualenv:
$ git clone git://github.com/schmichael/mmstats.git $ cd mmstats $ virtualenv . $ source bin/activate $ python setup.py develop $ ./run_flask_example # This starts up a sample web app $ curl http://localhost:5001/ $ curl http://localhost:5001/500 $ curl http://localhost:5001/status $ # If you have ab installed: $ ab -n 50 -c 10 http://localhost:5001/
Now to view the stats run the following in a new terminal:
$ # To get a raw view of the data: $ slurpstats mmstats-* $ # Or start up the web interface: $ mmash $ # Run pollstats while ab is running: $ pollstats -p flask.example. ok,bad,working mmstats-*
To cleanup stray mmstats files: rm mmstats-flask-*
The web interface will automatically reload when you change source files.
Put static files into static/ and template files into templates/
TODO: Factor mmash out into it’s own project (with a meaningful name?)
Testing
Feel free to use your favorite test runner like nose or pytest or just run:
$ python setup.py test
History
0.6.2 “Graphtastic” released 2012-03-23
Added live graphing of numeric metrics thanks to @haard’s work at PyCon
Documentation improvements
0.6.1 “MANIFEST.out” released 2012-03-08
Fix packaging issue
0.6.0 “PyCon 2012” released 2012-03-08
[API CHANGE] - MovingAverageField’s kwarg changed from window_size => size
Refactored __init__.py into fields, models, and default (and imported public bits into __init__)
Added TimerField (MovingAverageField + context manager)
Added docs (don’t get too excited, just a start)
0.5.0 “100% More Average” released 2012-02-25
[API CHANGE] - RunningAverage field is now AverageField
Added MovingAverageField with window_size=100 parameter
Tests can now be run via “python setup.py test”
0.4.1 “Derpstats” released 2012-01-31
Fixed pollstats
Updated README slightly
0.4.0 “On the Road to Pycon” released 2012-01-17
Added clean module and cleanstats script to clean stale mmstat files
Added path kwarg to MmStats class to allow easy path overriding
Added StringField for UTF-8 encoded strings
Added StaticFloatField & StaticDoubleField
Added created UNIX timestamp (sys.created) to default MmStats class
Moved all modules into mmstats package
Fixed mmash template packaging
Fixed test mmstat file cleanup
Refactored reading code into mmstats.reader module
0.3.12 “Meow” released 2011-11-29
Use ctypes.get_errno() instead of Linux specific wrapper
0.3.11 “Rawr” released 2011-11-29
Fix libc loading on OSX
0.3.10 “π²” released 2011-11-28
PyPy support (switched from ctypes._CData.from_buffer to .from_address)
Multiple calls to MmStats().remove() no longer error (makes testing easier)
0.3.9 “MLIT” released 1970-01-01
Mistag of 0.3.8
0.3.8 “Hapiness” released 2011-11-20
Allow filename templating with %PID% and %TID% placeholders
Allow setting filename template via MMSTATS_FILES environment variable
Improved docs slightly
Fixed Ctrl-Cing run_flask_example script
Fixed 64 bit integer fields on 32 bit platforms
Fixed StaticInt64Field (was actually a uint64 field before)
Fixed slurpstats debug output (always showed first 40 bytes of file)
Strip newlines from org.python.version
0.3.7 “Depressive Realism is for Winners” released 2011-11-17
Add pollstats utility (similar to dstat/vmstat)
Cleanup development/testing section of the README
Slight improvements to basic flask integration example
0.3.6 “The M is for Mongo” released 2011-11-09
Allow setting the value of CounterFields
0.3.5 “Ornery Orangutan” released 2011-10-20
Added a running average field
Made mmash more configurable and added a console entry point
Updated TODO
TODO
Add API to dynamically add fields to MmStat classes
Percentiles
Time based windows for moving averages (eg last 60 seconds)
Multiple exposed fields (average, mean, and percentiles from 1 model field)
Add alternative procedural writer API (vs existing declarative models)
Test severity of race conditions (especially: byte value indicating write buffer)
Test performance
Vary filename based on class name
Scrapped Ideas
Compounds Fields where 1 Writer Field = Many Mmap/Reader Fields
This seemed like a honking great idea at first. Compound fields would look just like a mini-MmStat model:
class SamplingCounterField(CompoundField): """Records increments per ms every N increments""" counter = CounterField() per_ms = UInt64Field() class _Counter(object): """Implement counter/rate-sampling logic here""" def __get__(self, inst, owner): if inst is None: return self return inst._fields[self.key]._counter_instance
The blocker is that there’s no way to atomically update all of the compound fields. The only way to accomplish this is for compound fields to appear as a single double buffered field with each component field as a type in the type signature:
class SamplingCounterField(DoubleBufferedField): initial = ( CounterField.initial, UInt64Field.initial, ) buffer_type = ( CounterField.buffer_type, UInt64Field.buffer_type, ) type_signature = ( CounterField.type_signature + UInt64Field.type_signature )
Obviously an actual implementation should remove the redundant references to the component types.
Note: Lack of atomicity is not a blocker for exposing fields such as Mean, Median, and Percentiles.
Solution: Future versions of the mmstats format should support structs as values instead of just scalars so that a single write buffer offset can point to multiple values.
Metadata metaprogramming
To get around having to dynamically creating the structs due to a variable label size, put the labels in a header index with a pointer to the actual struct field.
Page Flipping
Store metadata seperate from values. Then store values in multiple pages and flip between pages for read/write buffering.
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.