Stat publishing and consuming tools
Project description
About
Mmstats is a way to expose (mmstats.py) and read (slurpstats, pollstats, mmash) diagnostic/statistical values for applications.
Think of mmstats as /proc for your application and the readers as procps utilities.
Goals
Separate publishing/writing from consuming/reading tools
Platform/language independent (a Java writer can be read by a Python tool)
Predictable performance impact for writers via:
No locks (1 writer per thread)
No syscalls (after instantiation)
All in userspace
Reading has no impact on writers
Optional persistent (writer can sync anytime)
1-way (Publish/consume only; mmstats are not management extensions)
Usage
Requirements
CPython 2.6 or 2.7 (Windows is untested)
PyPy (only tested in 1.7, should be faster in 1.8)
Using
python setup.py install # Or copy mmstats.py into your project
import mmstats
Create a subclass of mmstats.MmStats like:
class WebStats(mmstats.MmStats): status2xx = mmstats.UIntField(label='status.2XX') status3xx = mmstats.UIntField(label='status.3XX') status4xx = mmstats.UIntField(label='status.4XX') status5xx = mmstats.UIntField(label='status.5XX')
Instantiate it once per thread/process:
webstats = WebStats(label_prefix='web.stats.')
Record some data:
if response.status_code == 200: webstats.status2xx += 1
Run slurpstats to read it
Run mmash to create a web interface for stats
Run pollstats -p web.stats.status 2XX,3XX,4XX,5XX /tmp/mmstats-* for a vmstat/dstat like view.
Did a process die unexpectedly and leave around a stale mmstat file? cleanstats /path/to/mmstat/files will check to see which files are stale and remove them.
Development
It’s easiest to develop mmstats within a virtualenv:
$ git clone git://github.com/schmichael/mmstats.git $ cd mmstats $ virtualenv . $ source bin/activate $ python setup.py develop $ ./run_flask_example # This starts up a sample web app $ curl http://localhost:5001/ $ curl http://localhost:5001/500 $ curl http://localhost:5001/status $ # If you have ab installed: $ ab -n 50 -c 10 http://localhost:5001/
Now to view the stats run the following in a new terminal:
$ # To get a raw view of the data: $ slurpstats mmstats-* $ # Or start up the web interface: $ mmash $ # Run pollstats while ab is running: $ pollstats -p flask.example. ok,bad,working mmstats-*
To cleanup stray mmstats files: rm mmstats-flask-*
The web interface will automatically reload when you change source files.
Put static files into static/ and template files into templates/
TODO: Factor mmash out into it’s own project (with a meaningful name?)
Testing
Run your favorite Python test runner (py.test or nosetests)
Run slurpstats.py
Clean /tmp/mmstats-* files up
Data Structures
There are two types of data structures so far in mmstats:
buffered
unbuffered
Buffered structures use multiple buffers for handling values which cannot be written atomically.
Unbuffered structures have ff in the write buffer field.
Buffered
label size |
label |
type size |
type |
write buffer |
buffers |
---|---|---|---|---|---|
ushort |
char[] |
ushort |
char[] |
byte |
varies |
The buffers field length = sizeof(type) * buffers.
The current write buffer is referenced by: write_buffer * sizeof(type)
TODO: field for total number of buffers?
Unbuffered
label size |
label |
type size |
type |
write buffer |
value |
---|---|---|---|---|---|
ushort |
char[] |
ushort |
char[] |
ff |
varies |
The value field length = sizeof(type).
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.