This is a pre-production deployment of Warehouse, however changes made here WILL affect the production instance of PyPI.
Latest Version Dependencies status unknown Test status unknown Test coverage unknown
Project Description

plumd

What is plumd?

Plumd is an extensible metrics collection tool that implements a reader/writer plugin framework designed from the start to support dimensional metrics.

Built-in Plugins

  • Linux /proc reader for detailed system metrics
  • Nginx stub_status reader
  • Memcache stats reader
  • Redis info reader including per slave and db metrics
  • PHP-FPM reader including per php-fpm pool metrics, apc and opcache metrics
  • MySQL reader for basic status and table metrics (manual install of pymysql required)
  • Graphite writer

Current Status

Consider Plumd beta quality.

That said, it is stable and does not consume any significant amount of resources even when the Graphite servers are down for extended periods.

The main rough areas are:

  • default values for the Linux /proc readers need tuning
  • potential issues with handling of counters wrapping
  • unit tests for plugin related code need to be created
  • metric names, plugin names, etc are likely to change until v1.0 is released

Requirements

Plumd is designed for Linux like systems and runs on Python 2.7, 3.x and PyPy. The only dependency is PyYaml.

Quickstart

  • pip install plumd
  • create: /etc/plumd.yaml - see test.main.yaml
  • create: /etc/plumd.conf.d/proc.yaml - see proc.yaml
  • create: /etc/plumd.conf.d/graphite.yaml - see graphitetcp.yaml
  • test: plumd –config /etc/plumd.yaml
  • configure plumd to run from your favorite process manager
  • note: plumd does not require root access

Design

Plumd has been designed with several requirements in mind. It must:

  • Be reasonably lightweight
  • Support dimensional metrics
  • Be easily extended to read new metrics and write to new backends
  • Support routing metrics to different backends, depending on configuration

The system runs in a single process and uses threads to isolate each plugin. A typical instance consumes ~14mb of memory and ~0.5% cpu during polls on most systems.

Creating a reader plugin is as simple as sub-classing plumd.Reader and overriding the poll() method. To use it install your plugin on the system, create a yaml configuration and restart plumd.

You can instantiate a given plugin multiple times by creating multiple configurations for it with different parameters. Readers by default write to all loaded Writers however can be configured to write to specific Writer(s).

Planned for v1.0

  • generic log parser (eg. nginx access logs)
  • add llen, zcard, scard, etc metrics to redis for list of key prefixes (eg. if you’re using redis as a queue, ability to record queue sizes based on {‘key_type’: [‘key*’, ‘pre*’, ‘fixes*’]})
  • code review and reorg - code ‘works’ however needs some cleanup/
  • documentation updates
  • fixup defaults in linux proc, mysql readers - prune metrics to record, move to gauges/rates as appropriate
  • add functionality for plugins to enable/disable themselves on startup (eg. disable conntrack reader if iptables is not running)
  • metadata handling: raise exception on conflicts
  • investigate possible issues with counter wraps in rate calcs
  • flake8 / formatting

Planned Features / Changes - Short term

  • installer helper - create configuration under specified directory and populate configuration files with defaults from all built-in and configured plugins
  • add functionality for plugins to report all available metrics
  • improve mysql reader, add calculated values, others

Potential Features / Changes - Long Term

  • feature: writer plugins: statsd, opentsdb, influxdb, elastic, message queues, others
  • feature: server plugins for eg. Prometheus compatible endpoint

Credits

This package was created with Cookiecutter and the audreyr/cookiecutter-pypackage project template.

History

0.12 (2016-09-14)

  • NOTE: BREAKING CHANGES: plumd.plugins.readers/writers have moved to plumd.readers/writers - all plugin .yaml configurations will need to be updated with the new locations
  • reorganized code, reduced use of packages to only reader and writer folders
  • added maxmem metric to Redis reader using “config get maxmemory” call
  • removed reconnect/retry from Redis reader - it will now reconnect on next poll
  • added commandstats metrics to Redis reader
  • added llen/zcard/scard/pfcount capability to Redis reader for configured globs
  • added very limited Redis client to support new Redis metrics

0.11.1 (2016-09-07)

  • fixup bug in parsing network interface names from /proc/net/dev on Centos 6

0.11 (2016-09-06)

  • fixup bug in disk reader - re match on the mounted path, not the mounted device

0.10 (2016-09-04)

  • added unix domain socket to redis, memcache and php-fpm readers
  • added checks for DiskStats, LoadAverage /proc/diskstats file format
  • Conntrack reader no longer raises exception if /proc files are missing (eg. iptables not running)
  • fixed format bug in Plugin __str__ and __repr__ functions
  • added exception handling, poll() return checks and logging to Proc
  • NetDev reader now parses /proc/net/dev for metric names instead of using hard coded configurable values
  • added basic mysql reader for show global status only - requires pymysql which is not installed by default/dependancy
  • added ‘show table status’ recording of table metrics to mysql reader

0.9.1 (2016-08-28)

  • fixed bug in php-fpm reader - the bug was only triggered on non-default configs

0.9 (2016-08-28)

  • added PHP-FPM reader with builtin fastcgi client to check php-fpm, apc and opcache metrics
  • added accompanying cacheinfo.php script

0.8 (2016-08-25)

  • added Nginx reader
  • added db metrics to Redis reader (keys, avg ttl, etc)

0.7 (2016-08-25)

  • added very rough Memcache reader
  • added very rough Redis reader
  • fixed cpu metrics - values needed to be rates not gauges

0.6 (2016-08-24)

  • cpu metrics now in jiffys, removed percent calculations

0.5 (2016-08-21)

  • documentation updates
  • added conntrack proc reader
  • added vmstat proc reader
  • split proc reader into smaller classes
  • added some more ~sane defaults to various proc readers (more tuning required here)
  • fixes for /proc readers on very old kernels (eg. centos5)
  • potential performance improvements for a number of linux.proc readers
  • fixed bug in /proc/stat reader that would report cpu<n> metrics when per_cpu was False
  • logging format string is now configurable as ‘log.format’ - useful for eg. if running under systemd
  • added support for python3
  • note: commit 2f3d56c7df877a747e4841b8f88cbd2bd677b8f9 broke python3 support due to str.translate() calls (now fixed)
  • added basic unit tests
  • added concept of gauges and rates to appropriate proc readers - now possible to configure metrics to record as gauges or rates (per second)
  • proc readers: remove configured metrics if not found in various /proc files - readers will print errors re: missing metrics only once
  • removed plumd.plugin.reader and plumd.plugin.writer proxy objects - moved threads directly into plugin objects - slight improvement on cpu utilization - eg. 1 second polling of 300+ metrics uses spikes of ~7% max of an idle 500mhz celeron cpu according to top (3s polling interval)

0.4 (2016-08-01)

  • added /proc readers for: cpu, diskspace, diskusage, interrupt (overall, not per interupt), ip, loadavg, memory, network, proc, sockstat, tcp, udp, uptime, vmstat
  • shutdown handling: ensured writer plugins given a configurable time to flush on shutdown, reworked reader/writer/graphite code to ensure clean shutdown
  • conf object: added recursive searching for configuration files (easier for configuration management)
  • graphite writer: add config/flag for including/excluding meta data tag names in the metric
  • graphite writer: on shutdown flush partial metrics buffer
  • graphite writer: fixes for various connection refused bugs, logic fixes related to no connection while shutting down
  • modified Render class, it is now an iterator that returns full chunks of rendered metrics
  • removed influxdb writer due to Render changes - may add it again in the future
  • removed Graphite Pickle rendered - may add it again in the future
  • updates to Graphite writer for Render changes
  • general code review, cleanup
  • documentation updates
  • feature: add hostname to metrics metadata (set meta.hostname=True in config)
  • influxdb writer: review how metrics are handled, eg. cpu idle value=0.0 => cpu times idle=0.0,system=10.1,user=89.9 etc
  • config.get - support specifying the type to return via get call eg. self.config.get(“blah”, int)
  • feature: timezones - set timezone in main configuration
  • cleaned up debug code, currently too many debug calls
  • added __str__ and __repr__ for classes that make sense - for debugging
  • changed internal metric handling to support back ends that handle multiple values in one measurement (eg. influxdb, cpu idle=10.0,user=80.0,system=10.0)
  • removed metadata from each metric class (eg. Int, Float, etc) and created Meta class to handle arbitrary metadata
  • created Result class to encapsulate multiple values as well as a single set of metadata
  • updated ResultSet to use Result

0.3.3 (2016-01-08)

  • switched from gevent to standard threads for concurrency - one less dependency, now fewer restrictions developing plugins
  • optimized metric rendering - pass a generator into the render functions instead of calling render function in a loop
  • changed Render get_chunk to get_batch, changed return from a generator to a list so writer plugins can see if there are metrics to process
  • elasticsearch writer plugin now uses requests instead of elasticsearch client - one less dependency
  • influxdb writer now uses requests instead of influxdb client - one less dependency
  • added a cycle of (url, session) to elasticsearch writer to cycle through each elasticsearch server defined
  • fixed spurious logging in writers from urllib3
  • added retries to elasticsearch writer
  • added traceback logging on each call to onstart/onstop/poll/push and prints to log to make debugging plugin issues easier
  • removed writer plugin metadata - only supporting host and metric metadata
  • reader configuraton can now specify a list of writer plugins to send to - don’t specify anything to send metrics to all loaded writers
  • fixed bug in Interval - __enter__ did not return self
  • added remaining and elapsed property methods to Interval to see loop times mid loop
  • added event to Interval class to cause it to wait on the event instead of time.sleep - allows clean and quick shutdown of all threads on signal
  • added block=True and timeout to reader when pushing to writer queue - it’s ok to block for the remaining time in poll loop
  • host level metadata added to metrics - writer plugins need to add it to each metric when rendering
  • elastic writer - use /<index>/<type>/_bulk url instead of /_bulk
  • influxdb writer - fixed line formatting, rough key escaping, metric naming/formatting fixed, http and udp writers updated and working
  • influxdb, elastic writers - added timeouts to http requests
  • elastic (bulk api), influxdb (udp, http) and graphite (udp, tcp) writer plugins are now functional however need testing - there are likely several bugs in each

0.3.2 (2016-01-05)

  • several quantifiedcode fixes
  • changed string templating from “%s”%(str) to “{0}”.format()
  • added missing doc strings
  • renamed classes to be more pythonic
  • renamed conf.set function
  • removed wildcard import in __main__
  • removed default args from plugins, metrics, others
  • added default capability to plugin configs - just define defaults dict in class and they get loaded automatically into self.config
  • added enabled: flag in plugin configurations to control loading of plugins - no more moving files around
  • added graphite writer w/ plaintext and pickle support - partially implemented
  • added influxdb writer - partially implemented
  • added elasticsearch writer - mostly functional, uses bulk api, needs improved handling of metadata
  • added ResultSet to allow plugins to return multiple measurements
  • added Render class to allow writer plugins to subclass and receive metrics in chunks pre-formatted for submission to their backend
  • added Int, Float, String, Boolean, Counter, Timer, Gauge, etc classes to encapsulate a recorded metric value

0.3.1 (2016-01-03)

  • removed bridge object
  • reworked preader and pwriter - readers now write directly to writers
  • check in preader to ensure poll() returns plumd.record class
  • renamed plumd.manager to load
  • misc bug fixes

0.3.0 (2016-01-1)

  • switched to gevent Queues from 0MQ for metrics passing
  • removed bridge sub process/cd
  • switched from string metrics to plumd.record class w/ named tuples and tags
  • reworked bridge, read and write - improved shutdown/error handling
  • changed configuration/error names to be more intuitive, doc updates
  • updated psutil readers/others to support new metrics
  • misc bug fixes
  • found issue w/ module folder named .io (likely why build is failing)

0.2.0 (2016-01-1)

  • fixed module imports
  • added very rough psutil based readers for cpu, disk, diskio, mem, net and swap

0.1.0 (2016-01-1)

  • First release on PyPI.
Release History

Release History

0.12.1

This version

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.12

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.11.1

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.10

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.9.1

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.9

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.8

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.7

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.5.2

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.5.0

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.4.0

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.3.3

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.3.2

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.3.1

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.3.0

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

0.2.0

History Node

TODO: Figure out how to actually get changelog content.

Changelog content for this version goes here.

Donec et mollis dolor. Praesent et diam eget libero egestas mattis sit amet vitae augue. Nam tincidunt congue enim, ut porta lorem lacinia consectetur. Donec ut libero sed arcu vehicula ultricies a non tortor. Lorem ipsum dolor sit amet, consectetur adipiscing elit.

Show More

Download Files

Download Files

TODO: Brief introduction on what you do with files - including link to relevant help section.

File Name & Checksum SHA256 Checksum Help Version File Type Upload Date
plumd-0.12.1-py2.py3-none-any.whl (77.0 kB) Copy SHA256 Checksum SHA256 py2.py3 Wheel Sep 14, 2016
plumd-0.12.1.tar.gz (78.6 kB) Copy SHA256 Checksum SHA256 Source Sep 14, 2016

Supported By

WebFaction WebFaction Technical Writing Elastic Elastic Search Pingdom Pingdom Monitoring Dyn Dyn DNS HPE HPE Development Sentry Sentry Error Logging CloudAMQP CloudAMQP RabbitMQ Heroku Heroku PaaS Kabu Creative Kabu Creative UX & Design Fastly Fastly CDN DigiCert DigiCert EV Certificate Rackspace Rackspace Cloud Servers DreamHost DreamHost Log Hosting